mmormota
Posts: 10
Joined: Fri Aug 05, 2016 10:17 pm

streaming, missing frames

Mon Aug 08, 2016 1:53 pm

I am about to improve a video streaming application (wifibroadcast). I like to keep as low latency as possible, but avoid missing frames.
The situation is a bit complicated.

The sender part is a Pi with camera, raw h.264 stream at 48fps. No sync info added to the stream. There are key frames, so the size of frames are very different, so as the delay of arrival on the receiver side.

The receiver side is another Pi with a hello_video program, modded to read from stdi instead of a file. The hdmi works at 60fps. So there is an inherent flicker, every 4th frame has to be repeated. (5*48 = 4*60)

My problem is the stutter because of missing frames. I mean, some frames are missing from the displayed series. Allways just one missing frame, never two in series, but pretty often. ( I carefully analysed the displayed video using a 240 fps camera, checking the recording frame by frame)

I suspect (maybe not correct) that the problem is as follows:
- some frames are encoded into much more bytes then others
- the delivery time from sender to receiver on a slow channel depends on the amount of bytes
- so a "big" frame comes a bit late, and a "small" frame follows it very close, so the decoder passes it to the render before it passes the previous one to the hdmi
- the render drops the incomplete previosu frame and passes the new one to the hdmi
- this way a frame lost

My question:
Is it possible to avoid the frame dropping? For example force the render to finish a frame before accepting a new one?

As proof of concept I did the following: I set the playback framerate to 48fps. I added a small delay (100msec) to the start of the stdin capture loop. This way the stream filled the stdout->stdin fifo for 100msec, then the decode-render-hdmi process started.
The video was smooth without missing frames. Unfortunately it's not a solution, because the 100msec additional latency slightly drifts because of the inaccurate crystals. Either increases toward unacceptable latency or inversely it vanishes and stuttering comes back.
I am thinking about a feedback, controlling the playback speed in order to keep constant latency. However it is hard, as there is one way communication without timing info. The amount of the data in the pipelines has big jitter, and I don't even know how to get the number of bytes in the pipe for averaging.
So I concluded that the ideal way is controlling somehow the pipe toward the render, not allow to pass a new frame before the previous is ready and delivered to the hdmi port. This way the latency is minimal too.

Is it possible, if yes, how to do that?

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 4531
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: streaming, missing frames

Mon Aug 08, 2016 3:40 pm

If your pipe is going to take N ms to transport the biggest packet, then your only option is to delay all frames by N ms. Any other solution is going to result in jitter and/or dropped frames.
video_scheduler/clock should be able to do that for you, although you'll then need to arrange to have timestamps on your frames, and you may need to account for clock drift.

video_render as a component will just render frames as they come in. If a second buffer is presented before the VSync has occurred, then the first buffer will be dropped. It is working to as low a latency as possible.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
Please don't send PMs asking for support - use the forum.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

mmormota
Posts: 10
Joined: Fri Aug 05, 2016 10:17 pm

Re: streaming, missing frames

Mon Aug 08, 2016 4:39 pm

Thank you for the fast and informative reply.
If your pipe is going to take N ms to transport the biggest packet, then your only option is to delay all frames by N ms.
Sure, I agree.
Any other solution is going to result in jitter and/or dropped frames.
Consider the stream is very unic, there is a 48fps source, and a 60fps receiver. So the receiver part is way faster, no objective reason to drop frames.
If I record a 48fps video to a file, and play that file with hello_video, there are no missing frames. Of course there is some jitter, because every 5th frame is repetition, I mean: 1,2,3,4,4,5,6,7,8,8,9,10,11,12,12,13,14 etc.

What I am about to achieve is a similar relative smooth video but from living async stream. Considering the slow crystal drift, it is natural that sometimes there is a part with more or less then 4 consecutive frames between repeated doubles, but it's acceptable.

Right now there are missing frames as well. For example:
_18, 19, 20, 21, 22, _24, 25, 26, 27, 28, 29, 30, _32, 33, 34, 35, 36, 37, _39, _41, 42, _44, 45, 46, 47, _49, 50, 51, 52, 53, 54, _56
I think it's possible to avoid, just hard to figure out how to do that without deep knowlidge in the Broadcom components.

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 4531
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: streaming, missing frames

Mon Aug 08, 2016 7:57 pm

Not with video_render unless you do something clever before you present the images to it.

Go and look up the DispmanX API (eg https://github.com/raspberrypi/firmware ... dispmanx.c, or https://github.com/AndrewFromMelbourne/ ... i/DispmanX), and knock yourself out doing it yourself. DispmanX can give you a callback on vsync, and also when the previous image has been removed from the display.

Any mechanism that is not tunnelling OMX buffers has a performance penalty, as images have to be converted from an internal format to planar, copied to the ARM, and then back to the GPU.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
Please don't send PMs asking for support - use the forum.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

mmormota
Posts: 10
Joined: Fri Aug 05, 2016 10:17 pm

Re: streaming, missing frames

Fri Aug 19, 2016 2:30 pm

All I need is smooth video without frame dropping, I hope it's possible with some modification on the hello_video sample. As there are 48 input and 60 output frames per sec, there is no real reason to drop input frames.

My first idea based on that hello_video plays smooth from a file, I mean if the raw h.264 is recorded first to a file, file playback is smooth. If it plays the live stream, it stutters. File playback keeps smooth if I control the speed using xScale too.

So my idea was controlling the speed based on the content of the fifo, this way keep a more or less constant latency. Too much bytes in the fifo -> increase speed and vica versa.
It is almost working. ;)
It is smooth until a real dropout (wifi lost signal for a moment).
Then it doesn't interested in the xScale value any more, just pushes out frames asap, and drops if a new frame is ready before vsync. The latency keeps on minimum, and stutter is back with frequent missing frames.
Same happens if I restart either the Tx or Rx while keep working the other.
I suspect it depends somehow on the timestamps. The clock component lost synch, and chooses another working mode: "don't care timing, push out frames asap."
Unfortunately I don't understand how the clock component is working, and have no idea how to avoid this effect.

Please, if you have any idea how to avoid dropped frames stutter, please point me to the right direction.

I have a different idea too, not yet tried:
- separete the frames, and fill a single frame to the input buffer, and pass it to the decoder
- wait a few msecs to avoid too early decoding and rendering, then pass the next frame
- this way maybe possible to avoid that the frame is decoded&rendered too early and override the previous before vsync
Has it a chance to work properly? I think it depends on how fast is the gpu processing, that allows such a delay.

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 4531
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: streaming, missing frames

Fri Aug 19, 2016 2:59 pm

Two options I can see for you:
1 - The DispmanX API as I have already pointed out.
2 - Do NOT tunnel between video_decode and video_render. Take the buffers back to the client and arrange a FIFO system so that video_render is only ever allowed 2 buffers at once. The buffer that was on the display will only be returned at the point it has been replaced by the new buffer, so you can guarantee every frame is on the display for at least one frame. Up to you how you deal with the potential situation of the incoming frame rate being faster than the output one. You will get a stutter every time your large I frames get delayed due to transmission delays, but that is inherent in not accounting for your badnwidth limited link with some form of buffering.

Trying to control anything at the input side of video_decode is likely to fail.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
Please don't send PMs asking for support - use the forum.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

mmormota
Posts: 10
Joined: Fri Aug 05, 2016 10:17 pm

Re: streaming, missing frames

Fri Aug 19, 2016 3:08 pm

Thank you very much for the fast reply.
DispmanX seems rather hard, so I try the other option.

mmormota
Posts: 10
Joined: Fri Aug 05, 2016 10:17 pm

Re: streaming, missing frames

Mon Aug 22, 2016 5:14 am

The buffer that was on the display will only be returned at the point it has been replaced by the new buffer, so you can guarantee every frame is on the display for at least one frame.
1. You mean each output buffer holds exactly a single frame?
How can I do that? Can I order somehow the decode component to put each frame to a separate buffer? Or I have to feed the decoder with a single frame, wait for callback, repeat?
2. When the render component gives back the buffer, it guarantee the frame hits the hdmi output, or still there is the possibility that a new frame from the next buffer overrides it?
3. Is there a way to get vsync callback (using ilclient and omx)?
4. The output buffer of the decode component can be shared as input buffer for the render? If sharing the buffer is possible, how to do that? Or memcpy() required? (a few lines of code would be really helpful)

Sorry for these lamer questions, but the specs are unclear for me. I spent many hours searching for specs and sample code, but still in the mist.

manish21thakur
Posts: 4
Joined: Mon Aug 22, 2016 7:12 am

Re: streaming, missing frames

Mon Aug 22, 2016 7:27 am

I tried downloading a broadcasting things on twitch because Steam didn't work and mine wasn't compatible to go with Twitch so I need help on this .

mmormota
Posts: 10
Joined: Fri Aug 05, 2016 10:17 pm

Re: streaming, missing frames

Tue Aug 23, 2016 6:26 pm

I am collecting info for the correct solution, but there are still some unclear points (my previous post).
Have some success with the input side trial, no missing frames. Unfortunately it's not a general solution but tuned to the special framerates.
- removed the clock and scheduler component, piped the decoder to the render
- search for the frame delimiters on the input side
- feed the decoder with whole frames with schedule

mmormota
Posts: 10
Joined: Fri Aug 05, 2016 10:17 pm

Re: streaming, missing frames

Thu Aug 25, 2016 4:08 am

DispmanX API provides a vsync callback, now the feed of the decode-render chain keeps sync with hdmi vsync. Thank you for the DispmanX idea.
I checked several hundred numbered frames (using a 240fps recording) - no missing frames.

IM2017
Posts: 1
Joined: Thu Mar 30, 2017 5:42 pm

Re: streaming, missing frames

Thu Mar 30, 2017 5:46 pm

Hello mmormota!

Did you ever find a solution to the stutter problem?

Cheers

Return to “OpenMAX”

Who is online

Users browsing this forum: No registered users and 2 guests