huww98
Posts: 4
Joined: Thu Sep 06, 2018 2:05 pm

gstreamer v4l2h264enc/v4l2src issue. Not working/Very large latency/Freeze after a while

Thu Jun 20, 2019 5:25 am

Hi,
I'm building an application that can live stream from raspberry pi camera.

Previously, I've managed to stream using this pipeline:

Code: Select all

raspivid -t 0 -w 1296 -h 972 -fps 30 -b 4000000 --flush -o - | ./gst-launch-1.0 -v \
    rtpbin name=rtpbin \
    fdsrc do-timestamp=true ! h264parse ! rtph264pay config-interval=-1 ! rtprtxqueue ! rtpbin.send_rtp_sink_0 \
    rtpbin.send_rtp_src_0 ! udpsink host=${IP_OF_PC} port=5000 \
    udpsrc port=5005 ! rtpbin.recv_rtcp_sink_0
And on reciving side:

Code: Select all

gst-launch-1.0 -v \
    rtpbin name=rtpbin do-retransmission=true latency=70 \
    udpsrc port=5000 ! "application/x-rtp,media=video,clock-rate=90000,encoding-name=H264" ! rtpbin.recv_rtp_sink_0 \
    rtpbin. ! rtph264depay ! decodebin ! autovideosink \
    rtpbin.send_rtcp_src_0 ! udpsink port=5005 host=${IP_OF_PI} sync=false async=false
This works pretty well, it handles retransmission and overall latency is about 200ms.

Then, I heard of dmabuf from other posts, and want to try if this can improve performance (reduce latency) further. So I rpi-update to latest, cloned gstreamer cerbero, checkout 1.16.0 and run

Code: Select all

./cerbero-uninstalled bootstrap && ./cerbero-uninstalled package gstreamer-1.0

Then, after 8 hours of waiting, I can run gstreamer 1.16.0 in ./cerbero-uninstalled shell.

However, I'm facing some issues.

Code: Select all

GST_DEBUG=3 ./gst-launch-1.0 -v rtpbin name=rtpbin \
    v4l2src ! video/x-h264,width=1296,height=972,framerate=30/1,profile=high ! rtph264pay config-interval=-1 ! rtprtxqueue ! rtpbin.send_rtp_sink_0 \
    rtpbin.send_rtp_src_0 ! udpsink host=192.168.137.1 port=5000 sync=false \
    udpsrc port=5005 ! rtpbin.recv_rtcp_sink_0
This pipeline runs, but freeze after 1 minute or so, no more UDP packets sent, and a log is printed after several secends or at Ctrl+C
0:01:36.566698007 1259 0x14ceac0 WARN v4l2src gstv4l2src.c:915:gst_v4l2src_create:<v4l2src0> Timestamp does not correlate with any clock, ignoring driver timestamps.

Print screen with Win+Shift+S twice (I'm on Windows and ssh to raspberry pi using WSL) makes this easy to reproduce (strange enough), but it can also appear if i don't do this. Tried to insert a queue, not helping.

To study the timestamp problem, I set v4l2src do-timestamp=true and udpsink sync=true, then I get about 5 seconds latency. I don't know what's wrong.

For dmabuf, I test this with

Code: Select all

GST_DEBUG=4 ./gst-launch-1.0 -v v4l2src io-mode=4 ! video/x-raw,width=1296,height=972,framerate=30/1,format=I420 ! v4l2h264enc output-io-mode=5 ! video/x-h264,profile=high ! fakesink
It's not working.

0:00:00.278971825 1529 0x22f200 INFO v4l2bufferpool gstv4l2bufferpool.c:557:gst_v4l2_buffer_pool_set_config:<v4l2h264enc0:pool:src> increasing minimum buffers to 2
0:00:00.279044845 1529 0x22f200 INFO v4l2bufferpool gstv4l2bufferpool.c:570:gst_v4l2_buffer_pool_set_config:<v4l2h264enc0:pool:src> reducing maximum buffers to 32
0:00:00.279498384 1529 0x22f200 INFO v4l2bufferpool gstv4l2bufferpool.c:570:gst_v4l2_buffer_pool_set_config:<v4l2h264enc0:pool:src> reducing maximum buffers to 32
Redistribute latency...
0:00:00.279900672 1529 0x22f200 INFO v4l2 gstv4l2object.c:3713:gst_v4l2_object_set_format_full:<v4l2h264enc0:sink> Set output framerate to 30/1
0:00:00.280051817 1529 0x22f200 INFO v4l2 gstv4l2object.c:2938:gst_v4l2_object_setup_pool:<v4l2h264enc0:sink> accessing buffers via mode 5
0:00:00.280086661 1529 0xcd000 INFO GST_EVENT gstevent.c:1450:gst_event_new_latency: creating latency event 0:00:00.000000000
0:00:00.280383065 1529 0xcd000 INFO bin gstbin.c:2780:gst_bin_do_latency_func:<pipeline0> configured latency of 0:00:00.000000000
0:00:00.280622386 1529 0x22f200 INFO v4l2bufferpool gstv4l2bufferpool.c:557:gst_v4l2_buffer_pool_set_config:<v4l2h264enc0:pool:sink> increasing minimum buffers to 2
0:00:00.280745041 1529 0x22f200 INFO v4l2bufferpool gstv4l2bufferpool.c:570:gst_v4l2_buffer_pool_set_config:<v4l2h264enc0:pool:sink> reducing maximum buffers to 32
0:00:00.280854259 1529 0x22f200 INFO v4l2bufferpool gstv4l2bufferpool.c:586:gst_v4l2_buffer_pool_set_config:<v4l2h264enc0:pool:sink> adding needed video meta
/GstPipeline:pipeline0/v4l2h264enc:v4l2h264enc0.GstPad:sink: caps = video/x-raw, width=(int)1296, height=(int)972, framerate=(fraction)30/1, format=(string)I420, interlace-mode=(string)progressive, colorimetry=(string)bt601
/GstPipeline:pipeline0/GstCapsFilter:capsfilter0.GstPad:sink: caps = video/x-raw, width=(int)1296, height=(int)972, framerate=(fraction)30/1, format=(string)I420, interlace-mode=(string)progressive, colorimetry=(string)bt601
0:00:00.283563928 1529 0x22f200 INFO v4l2 gstv4l2object.c:3671:gst_v4l2_object_set_format_full:<v4l2src0:src> Set capture framerate to 30/1
0:00:00.283719135 1529 0x22f200 INFO v4l2 gstv4l2object.c:2938:gst_v4l2_object_setup_pool:<v4l2src0:src> accessing buffers via mode 4
0:00:00.284253142 1529 0x22f200 INFO v4l2bufferpool gstv4l2bufferpool.c:557:gst_v4l2_buffer_pool_set_config:<v4l2src0:pool:src> increasing minimum buffers to 2
0:00:00.284383714 1529 0x22f200 INFO v4l2bufferpool gstv4l2bufferpool.c:570:gst_v4l2_buffer_pool_set_config:<v4l2src0:pool:src> reducing maximum buffers to 32
0:00:00.284691524 1529 0x22f200 INFO v4l2bufferpool gstv4l2bufferpool.c:570:gst_v4l2_buffer_pool_set_config:<v4l2src0:pool:src> reducing maximum buffers to 32
0:00:00.295947590 1529 0x22f200 ERROR v4l2allocator gstv4l2allocator.c:944:gst_v4l2_allocator_alloc_dmabuf:<v4l2src0:pool:src:allocator> Failed to export DMABUF: Inappropriate ioctl for device
0:00:00.296192328 1529 0x22f200 ERROR v4l2bufferpool gstv4l2bufferpool.c:479:gst_v4l2_buffer_pool_alloc_buffer:<v4l2src0:pool:src> failed to allocate buffer
0:00:00.296300660 1529 0x22f200 WARN bufferpool gstbufferpool.c:305:do_alloc_buffer:<v4l2src0:pool:src> alloc function failed
0:00:00.296404722 1529 0x22f200 WARN bufferpool gstbufferpool.c:338:default_start:<v4l2src0:pool:src> failed to allocate buffer
0:00:00.296590346 1529 0x22f200 ERROR v4l2bufferpool gstv4l2bufferpool.c:914:gst_v4l2_buffer_pool_start:<v4l2src0:pool:src> allocate failed
0:00:00.296697949 1529 0x22f200 ERROR bufferpool gstbufferpool.c:559:gst_buffer_pool_set_active:<v4l2src0:pool:src> start failed
0:00:00.296846906 1529 0x22f200 WARN v4l2src gstv4l2src.c:662:gst_v4l2src_decide_allocation:<v4l2src0> error: Failed to allocate required memory.
0:00:00.297027895 1529 0x22f200 WARN v4l2src gstv4l2src.c:662:gst_v4l2src_decide_allocation:<v4l2src0> error: Buffer pool activation failed

Any idea how to resolve the "dmabuf Not working", "sync=true very large latency", or "freeze after a while" issue? Or should I just use raspivid for the live stream? raspivid seems a lot more stable.
And any idea how can I further reduce streaming latency?

Any help is appreciated.

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 6895
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: gstreamer v4l2h264enc/v4l2src issue. Not working/Very large latency/Freeze after a while

Thu Jun 20, 2019 7:54 am

Bcm2835-v4l2 (the camera v4l2 driver) does not support dmabufs. Currently it is using the vmalloc allocator, and those dmabufs would be incompatible with the codec anyway as it wants contiguous allocations.

With a simple tweak it should be possible to support dmabuf import into bcm2835-v4l2, so if the codec allocated the buffers that may then work. Unfortunately I don't believe there is a way to say that dmabuf import is supported but export isn't, so its unlikely to be a solution that can be committed.
Bcm2835-v4l2 could also be switched to use the contig allocator, but that requires memory allocation changes to increase the size of the cma heap.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

huww98
Posts: 4
Joined: Thu Sep 06, 2018 2:05 pm

Re: gstreamer v4l2h264enc/v4l2src issue. Not working/Very large latency/Freeze after a while

Fri Jun 21, 2019 3:50 pm

6by9 wrote:
Thu Jun 20, 2019 7:54 am
Bcm2835-v4l2 (the camera v4l2 driver) does not support dmabufs. Currently it is using the vmalloc allocator, and those dmabufs would be incompatible with the codec anyway as it wants contiguous allocations.

With a simple tweak it should be possible to support dmabuf import into bcm2835-v4l2, so if the codec allocated the buffers that may then work. Unfortunately I don't believe there is a way to say that dmabuf import is supported but export isn't, so its unlikely to be a solution that can be committed.
Bcm2835-v4l2 could also be switched to use the contig allocator, but that requires memory allocation changes to increase the size of the cma heap.
Thanks for your reply. So I will forget dmabuf for my project.

But if I use other io-mode, or just remove the io-mode settings, I get very bad performance (5 fps 1296*972). Is this expected? Or how can I further debug this?
With H264 retrieved directly from v4l2src, I can get 30fps, and very low latency. However, I want both raw and encoded stream.

Return to “Graphics, sound and multimedia”