Thank you for the fast and informative reply.
If your pipe is going to take N ms to transport the biggest packet, then your only option is to delay all frames by N ms.
Sure, I agree.
Any other solution is going to result in jitter and/or dropped frames.
Consider the stream is very unic, there is a 48fps source, and a 60fps receiver. So the receiver part is way faster, no objective reason to drop frames.
If I record a 48fps video to a file, and play that file with hello_video, there are no missing frames. Of course there is some jitter, because every 5th frame is repetition, I mean: 1,2,3,4,4,5,6,7,8,8,9,10,11,12,12,13,14 etc.
What I am about to achieve is a similar relative smooth video but from living async stream. Considering the slow crystal drift, it is natural that sometimes there is a part with more or less then 4 consecutive frames between repeated doubles, but it's acceptable.
Right now there are missing frames as well. For example:
_18, 19, 20, 21, 22, _24, 25, 26, 27, 28, 29, 30, _32, 33, 34, 35, 36, 37, _39, _41, 42, _44, 45, 46, 47, _49, 50, 51, 52, 53, 54, _56
I think it's possible to avoid, just hard to figure out how to do that without deep knowlidge in the Broadcom components.