dasl
Posts: 14
Joined: Fri Aug 28, 2020 4:05 am

video playback with gstreamer is choppy / laggy

Fri Apr 16, 2021 7:01 am

Hello, I am trying to use gstreamer to play a 1080p H264 video. When I play the video, the playback is slightly choppy / laggy.

Here is how I installed gstreamer and the required plugins:

Code: Select all

sudo apt-get install gstreamer1.0-tools gstreamer1.0-plugins-base-apps gstreamer1.0-plugins-good gstreamer1.0-plugins-bad gstreamer1.0-alsa
I have a test video which you may download here: https://www.dropbox.com/s/ayeb8816wii6d ... t.mp4?dl=0

Here we can see the test video is indeed 1080p h264 format:

Code: Select all

% ffprobe audio_video_sync_test.mp4
...
    Stream #0:0(und): Video: h264 (High) (avc1 / 0x31637661), yuv420p(tv, bt709), 1920x1080 [SAR 1:1 DAR 16:9], 295 kb/s, 30 fps, 30 tbr, 15360 tbn, 60 tbc (default)
...
And finally, I play the video with gstreamer:

Code: Select all

% gst-launch-1.0 -e -vvv filesrc location=audio_video_sync_test.mp4 ! qtdemux ! h264parse ! v4l2h264dec ! kmssink
Setting pipeline to PAUSED ...
Pipeline is PREROLLING ...
/GstPipeline:pipeline0/GstKMSSink:kmssink0: display-width = 640
/GstPipeline:pipeline0/GstKMSSink:kmssink0: display-height = 480
/GstPipeline:pipeline0/GstH264Parse:h264parse0.GstPad:sink: caps = video/x-h264, stream-format=(string)avc, alignment=(string)au, level=(string)4, profile=(string)high, codec_data=(buffer)01640028ffe1001e67640028acd100780227e5c05a808080a0000003002000000781e306224001000468ebef2c, width=(int)1920, height=(int)1080, framerate=(fraction)30/1, pixel-aspect-ratio=(fraction)1/1
/GstPipeline:pipeline0/GstH264Parse:h264parse0.GstPad:src: caps = video/x-h264, stream-format=(string)byte-stream, alignment=(string)au, level=(string)4, profile=(string)high, width=(int)1920, height=(int)1080, framerate=(fraction)30/1, pixel-aspect-ratio=(fraction)1/1, interlace-mode=(string)progressive, chroma-format=(string)4:2:0, bit-depth-luma=(uint)8, bit-depth-chroma=(uint)8, parsed=(boolean)true
/GstPipeline:pipeline0/v4l2h264dec:v4l2h264dec0.GstPad:sink: caps = video/x-h264, stream-format=(string)byte-stream, alignment=(string)au, level=(string)4, profile=(string)high, width=(int)1920, height=(int)1080, framerate=(fraction)30/1, pixel-aspect-ratio=(fraction)1/1, interlace-mode=(string)progressive, chroma-format=(string)4:2:0, bit-depth-luma=(uint)8, bit-depth-chroma=(uint)8, parsed=(boolean)true
/GstPipeline:pipeline0/v4l2h264dec:v4l2h264dec0.GstPad:src: caps = video/x-raw, format=(string)I420, width=(int)1920, height=(int)1080, interlace-mode=(string)progressive, multiview-mode=(string)mono, multiview-flags=(GstVideoMultiviewFlagsSet)0:ffffffff:/right-view-first/left-flipped/left-flopped/right-flipped/right-flopped/half-aspect/mixed-mono, pixel-aspect-ratio=(fraction)1/1, chroma-site=(string)mpeg2, colorimetry=(string)bt709, framerate=(fraction)30/1
/GstPipeline:pipeline0/GstKMSSink:kmssink0.GstPad:sink: caps = video/x-raw, format=(string)I420, width=(int)1920, height=(int)1080, interlace-mode=(string)progressive, multiview-mode=(string)mono, multiview-flags=(GstVideoMultiviewFlagsSet)0:ffffffff:/right-view-first/left-flipped/left-flopped/right-flipped/right-flopped/half-aspect/mixed-mono, pixel-aspect-ratio=(fraction)1/1, chroma-site=(string)mpeg2, colorimetry=(string)bt709, framerate=(fraction)30/1
Pipeline is PREROLLED ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
The playback is mostly ok, but as you can see it is occasionally a bit laggy. The same video plays without any lag with vlc or omxplayer. Playback with gstreamer uses only about 4% CPU, so I figure it must be using hardware decoding. Any idea what the issue is?

Here is info about my OS in case it is helpful (I'm just using the default OS)

Code: Select all

% uname -a
Linux piwall2 5.4.83-v7l+ #1379 SMP Mon Dec 14 13:11:54 GMT 2020 armv7l GNU/Linux
Thanks for looking!
Last edited by dasl on Fri Jun 18, 2021 8:52 am, edited 1 time in total.

drtechno
Posts: 246
Joined: Fri Apr 09, 2021 6:33 pm

Re: video playback with gstreamer is choppy / laggy

Thu May 13, 2021 10:00 pm

there are different types of buffering available, but at default, I don't think they set any.

check out the user manual for buffering with gstreamer: https://gstreamer.freedesktop.org/docum ... -buffering

dasl
Posts: 14
Joined: Fri Aug 28, 2020 4:05 am

Re: video playback with gstreamer is choppy / laggy

Sun Jun 06, 2021 5:14 am

drtechno wrote:
Thu May 13, 2021 10:00 pm
there are different types of buffering available, but at default, I don't think they set any.

check out the user manual for buffering with gstreamer: https://gstreamer.freedesktop.org/docum ... -buffering
I've liberally sprinkled usages of "queue2" into my pipeline. As I understand it, that should add some buffering into various parts of my pipeline (though I am not sure if buffering is needed in all the locations I added it). The choppy playback issue is still there unfortunately. I also tried the same with "queue".

See here for how I tested:

Code: Select all

% gst-launch-1.0 -e -vvv filesrc location=sync_test_1080p.mp4 ! qtdemux ! queue2 ! h264parse ! queue2 ! v4l2h264dec ! queue2 ! kmssink
Setting pipeline to PAUSED ...
Pipeline is PREROLLING ...
/GstPipeline:pipeline0/GstKMSSink:kmssink0: display-width = 720
/GstPipeline:pipeline0/GstKMSSink:kmssink0: display-height = 480
/GstPipeline:pipeline0/GstQueue2:queue2-0.GstPad:sink: caps = video/x-h264, stream-format=(string)avc, alignment=(string)au, level=(string)4, profile=(string)high, codec_data=(buffer)01640028ffe1001e67640028acd100780227e5c05a808080a0000003002000000781e306224001000468ebef2c, width=(int)1920, height=(int)1080, framerate=(fraction)30/1, pixel-aspect-ratio=(fraction)1/1
/GstPipeline:pipeline0/GstQueue2:queue2-0.GstPad:src: caps = video/x-h264, stream-format=(string)avc, alignment=(string)au, level=(string)4, profile=(string)high, codec_data=(buffer)01640028ffe1001e67640028acd100780227e5c05a808080a0000003002000000781e306224001000468ebef2c, width=(int)1920, height=(int)1080, framerate=(fraction)30/1, pixel-aspect-ratio=(fraction)1/1
/GstPipeline:pipeline0/GstH264Parse:h264parse0.GstPad:sink: caps = video/x-h264, stream-format=(string)avc, alignment=(string)au, level=(string)4, profile=(string)high, codec_data=(buffer)01640028ffe1001e67640028acd100780227e5c05a808080a0000003002000000781e306224001000468ebef2c, width=(int)1920, height=(int)1080, framerate=(fraction)30/1, pixel-aspect-ratio=(fraction)1/1
/GstPipeline:pipeline0/GstH264Parse:h264parse0.GstPad:src: caps = video/x-h264, stream-format=(string)byte-stream, alignment=(string)au, level=(string)4, profile=(string)high, width=(int)1920, height=(int)1080, framerate=(fraction)30/1, pixel-aspect-ratio=(fraction)1/1, interlace-mode=(string)progressive, chroma-format=(string)4:2:0, bit-depth-luma=(uint)8, bit-depth-chroma=(uint)8, parsed=(boolean)true
/GstPipeline:pipeline0/GstQueue2:queue2-1.GstPad:sink: caps = video/x-h264, stream-format=(string)byte-stream, alignment=(string)au, level=(string)4, profile=(string)high, width=(int)1920, height=(int)1080, framerate=(fraction)30/1, pixel-aspect-ratio=(fraction)1/1, interlace-mode=(string)progressive, chroma-format=(string)4:2:0, bit-depth-luma=(uint)8, bit-depth-chroma=(uint)8, parsed=(boolean)true
/GstPipeline:pipeline0/GstQueue2:queue2-1.GstPad:src: caps = video/x-h264, stream-format=(string)byte-stream, alignment=(string)au, level=(string)4, profile=(string)high, width=(int)1920, height=(int)1080, framerate=(fraction)30/1, pixel-aspect-ratio=(fraction)1/1, interlace-mode=(string)progressive, chroma-format=(string)4:2:0, bit-depth-luma=(uint)8, bit-depth-chroma=(uint)8, parsed=(boolean)true
/GstPipeline:pipeline0/v4l2h264dec:v4l2h264dec0.GstPad:sink: caps = video/x-h264, stream-format=(string)byte-stream, alignment=(string)au, level=(string)4, profile=(string)high, width=(int)1920, height=(int)1080, framerate=(fraction)30/1, pixel-aspect-ratio=(fraction)1/1, interlace-mode=(string)progressive, chroma-format=(string)4:2:0, bit-depth-luma=(uint)8, bit-depth-chroma=(uint)8, parsed=(boolean)true
/GstPipeline:pipeline0/v4l2h264dec:v4l2h264dec0.GstPad:src: caps = video/x-raw, format=(string)I420, width=(int)1920, height=(int)1080, interlace-mode=(string)progressive, multiview-mode=(string)mono, multiview-flags=(GstVideoMultiviewFlagsSet)0:ffffffff:/right-view-first/left-flipped/left-flopped/right-flipped/right-flopped/half-aspect/mixed-mono, pixel-aspect-ratio=(fraction)1/1, chroma-site=(string)mpeg2, colorimetry=(string)bt709, framerate=(fraction)30/1
/GstPipeline:pipeline0/GstQueue2:queue2-2.GstPad:sink: caps = video/x-raw, format=(string)I420, width=(int)1920, height=(int)1080, interlace-mode=(string)progressive, multiview-mode=(string)mono, multiview-flags=(GstVideoMultiviewFlagsSet)0:ffffffff:/right-view-first/left-flipped/left-flopped/right-flipped/right-flopped/half-aspect/mixed-mono, pixel-aspect-ratio=(fraction)1/1, chroma-site=(string)mpeg2, colorimetry=(string)bt709, framerate=(fraction)30/1
/GstPipeline:pipeline0/GstQueue2:queue2-2.GstPad:src: caps = video/x-raw, format=(string)I420, width=(int)1920, height=(int)1080, interlace-mode=(string)progressive, multiview-mode=(string)mono, multiview-flags=(GstVideoMultiviewFlagsSet)0:ffffffff:/right-view-first/left-flipped/left-flopped/right-flipped/right-flopped/half-aspect/mixed-mono, pixel-aspect-ratio=(fraction)1/1, chroma-site=(string)mpeg2, colorimetry=(string)bt709, framerate=(fraction)30/1
/GstPipeline:pipeline0/GstKMSSink:kmssink0.GstPad:sink: caps = video/x-raw, format=(string)I420, width=(int)1920, height=(int)1080, interlace-mode=(string)progressive, multiview-mode=(string)mono, multiview-flags=(GstVideoMultiviewFlagsSet)0:ffffffff:/right-view-first/left-flipped/left-flopped/right-flipped/right-flopped/half-aspect/mixed-mono, pixel-aspect-ratio=(fraction)1/1, chroma-site=(string)mpeg2, colorimetry=(string)bt709, framerate=(fraction)30/1
Pipeline is PREROLLED ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
Got EOS from element "pipeline0".
Execution ended after 0:01:02.996615519
Setting pipeline to PAUSED ...
Setting pipeline to READY ...
Setting pipeline to NULL ...
Freeing pipeline ...
Update:
I have also tried putting a queue or queue2 element in my pipeline, but it did not
help.

Here is what I tried:

gst-launch-1.0 -e -vvv filesrc location=audio_video_sync_test.mp4 ! qtdemux
! queue ! h264parse ! v4l2h264dec ! queue ! kmssink

I tried putting a queue or queue2 element after either the qtdemux or
v4l2h264dec elements, and also after both elements.

I also tried using "v4l2h264dec capture-io-mode=4" as recommended in one
forum post: viewtopic.php?f=67&t=240274

Here's a script with code demonstrating all of the testing I've done: https://gist.github.com/dasl-/1b26d1e81 ... 61b9316f29
Last edited by dasl on Thu Jun 10, 2021 4:32 am, edited 3 times in total.

dasl
Posts: 14
Joined: Fri Aug 28, 2020 4:05 am

Re: video playback with gstreamer is choppy / laggy

Tue Jun 08, 2021 11:40 pm

Here is a video of the graphics glitches that are occurring for me, in case it is enlightening for anyone: https://photos.app.goo.gl/b6FPuphAbMKdCgTMA

The glitches start happening at timestamp 0:42 and continue until the end of the video. The time at which the glitches occur appears to be fairly random. Perhaps more accurately these are video graphics glitches moreso than choppiness / lag.

dasl
Posts: 14
Joined: Fri Aug 28, 2020 4:05 am

Re: video playback with gstreamer is choppy / laggy

Fri Jun 18, 2021 9:09 am

Another update with another failed attempt to make things better:

The version of gstreamer I'd been working with thus far installed from the apt packages was an old version:

Code: Select all

% gst-launch-1.0 --version
gst-launch-1.0 version 1.14.4
GStreamer 1.14.4
http://packages.qa.debian.org/gstreamer1.0
So I tried installing the latest gstreamer version by building from source. Here is the script I used to build gstreamer from source on the raspberry pi: https://gist.github.com/dasl-/a43320008 ... a14f266a30

After running this script, I now had the latest gstreamer version:

Code: Select all

% LD_LIBRARY_PATH="${LD_LIBRARY_PATH}:/opt/gstreamer/lib/arm-linux-gnueabihf" /opt/gstreamer/bin/gst-launch-1.0 --version
gst-launch-1.0 version 1.19.1
GStreamer 1.19.1 (GIT)
Unknown package origin
I tried running all my tests again ( https://gist.github.com/dasl-/1b26d1e81 ... 61b9316f29 ), but the playback was still choppy.

Return to “Graphics programming”