Page 1 of 1

Using video clock as reference

Posted: Fri Jul 05, 2019 6:55 pm
by fd_
I'm trying to synchronise OpenMAX audio and video playback of a protocol that only has occasional audio data (The AirPlay mirroring protocol, to be specific). The protocol has a video stream that is periodically delivering video data packets, but only sends audio data when any audio is actually playing on the device.

Thus, I'm trying to use the video clock as the reference clock for the audio renderer. I've tried the obvious solution of setting OMX_IndexConfigTimeActiveRefClock with an eClock of OMX_TIME_RefClockVideo, but that doesn't seem to work. The video still only starts to display when audio playback is started, although my program has been feeding in video buffers before. Once audio data is supplied, I see the renderer fast-forward through all frames I fed in until it catches up with real-time.

Also, I found this post here on the forum that seems to suggest the Raspberry Pi's OpenMAX audio renderer implementation only supports acting as the clock master: https://www.raspberrypi.org/forums/view ... 9#p1453426. If this is really the case, I'm wondering how a program is supposed to use audio as the reference clock if there only is audio data occasionally?

Any help is highly appreciated!

Re: Using video clock as reference

Posted: Mon Jul 08, 2019 2:27 pm
by fd_
I eventually found the Broadcom audio render component has an OMX_IndexConfigBrcmClockReferenceSource configuration that can be used to tell it that it's not the clock master. I found it a bit strange that an extra configuration is necessary, but anyway, it seems to work :D

Here's the code I use for future reference:

Code: Select all

OMX_CONFIG_BOOLEANTYPE audio_is_clock_source;
memset(&audio_is_clock_source, 0, sizeof(OMX_CONFIG_BOOLEANTYPE));
audio_is_clock_source.nSize = sizeof(OMX_CONFIG_BOOLEANTYPE);
audio_is_clock_source.nVersion.nVersion = OMX_VERSION;
audio_is_clock_source.bEnabled = OMX_FALSE;
if (OMX_SetConfig(ilclient_get_handle(renderer->audio_renderer), OMX_IndexConfigBrcmClockReferenceSource,
    &audio_is_clock_source) != OMX_ErrorNone) {
    // Handle error here
}
I found the VMCS-X OpenMAX IL Components page as provided in the Raspberry Pi Github repo a great help once you understand the basic principles: http://htmlpreview.github.io/?https://g ... index.html

Re: Using video clock as reference

Posted: Mon Jul 08, 2019 3:55 pm
by 6by9
Sorry, I'd read your original post but got distracted before I replied.

audio_render doesn't really support taking an external clock as to do so requires audio resampling or clock slewing (just jumping to the relevant sample will sounds terrible with pops and clicks everywhere).
Because of that, video_render was always slaved to the clock, rather than being a clock master.

For your use case the only option is to tell audio_render not to act as clock master (as you've found), and for the clock not to wait for a start time on the audio port.

(Thanks for the github link - shame it only appears to work in Chrome (not Firefox). I'd always used http://www.jvcref.com/files/PI/document ... omponents/, or http://www.jvcref.com/files/PI/document ... index.html for MMAL. Those are updated manually by a 3rd party, but if github does it automagically for us then we can pull it into the docs).

Re: Using video clock as reference

Posted: Tue Jul 09, 2019 4:39 pm
by fd_
6by9 wrote:
Mon Jul 08, 2019 3:55 pm
Sorry, I'd read your original post but got distracted before I replied.
Never mind, thanks for all your detailed responses! :)
6by9 wrote:
Mon Jul 08, 2019 3:55 pm
audio_render doesn't really support taking an external clock as to do so requires audio resampling or clock slewing (just jumping to the relevant sample will sounds terrible with pops and clicks everywhere).
Because of that, video_render was always slaved to the clock, rather than being a clock master.

For your use case the only option is to tell audio_render not to act as clock master (as you've found), and for the clock not to wait for a start time on the audio port.
Yeah, I forgot to mention I'm also attaching video to the clock at port 80 and audio to 81, then setting the wait mask to wait for port 1 (which corresponds to 80 I think).
6by9 wrote:
Mon Jul 08, 2019 3:55 pm
(Thanks for the github link - shame it only appears to work in Chrome (not Firefox). I'd always used http://www.jvcref.com/files/PI/document ... omponents/, or http://www.jvcref.com/files/PI/document ... index.html for MMAL. Those are updated manually by a 3rd party, but if github does it automagically for us then we can pull it into the docs).
I just found that "hack" while trying to view the docs directly from your Github repository. Please note though that technically, this is not operated directly by Github. It's a GitHub user's project that kind of mis-uses the GitHub pages functionality in a clever way. It also seems to use third-party CORS proxies. But maybe you could make use of Github Pages for your firmware repository directly: https://pages.github.com/#project-site ?