Problem using Gstreamer with v4l2 and v4l2h264enc

[NKnuelle]

Hello everyone,

I am streaming the HDMI input of my Raspberry Pi 4B (4GB) via B101 module (tc358743) with Gstreamer to my local Nginx RTMP Server using gst-launch-1.0 and this pipeline:

Code: Select all

GST_DEBUG=2 gst-launch-1.0 flvmux streamable=true name=mux ! rtmpsink location="rtmp://localhost:1935/live/stream live=1" v4l2src io-mode=5 ! "video/x-raw,framerate=50/1,format=UYVY" ! v4l2h264enc output-io-mode=4 extra-controls="controls,video_bitrate=6000000;" ! video/x-h264,profile=high ! h264parse ! queue ! mux.

This is working fine, but I get some warnings:

Code: Select all

Leitung wird auf PAUSIERT gesetzt ...
0:00:00.099537468  1099  0x13fe5c0 WARN                    v4l2 gstv4l2object.c:4186:gst_v4l2_object_probe_caps:<v4l2h264enc0:src> Failed to probe pixel aspect ratio with VIDIOC_CROPCAP: Das Argument ist ungültig
0:00:00.100558298  1099  0x13fcc90 WARN                 v4l2src gstv4l2src.c:692:gst_v4l2src_query:<v4l2src0> Can't give latency since framerate isn't fixated !
0:00:00.100610650  1099  0x13fcc90 WARN              aggregator gstaggregator.c:1715:gst_aggregator_query_latency_unlocked:<mux> Latency query failed
Leitung ist aktiv und erfordert keinen VORLAUF …
Leitung wird auf ABSPIELEN gesetzt ...
New clock: GstSystemClock
Verzögerung neu verteilen …
0:00:00.170399490  1099  0x13f9db0 WARN          v4l2bufferpool gstv4l2bufferpool.c:790:gst_v4l2_buffer_pool_start:<v4l2h264enc0:pool:src> Uncertain or not enough buffers, enabling copy threshold
0:00:00.180015725  1099 0xb3a03200 WARN          v4l2bufferpool gstv4l2bufferpool.c:1189:gst_v4l2_buffer_pool_dqbuf:<v4l2h264enc0:pool:src> Driver should never set v4l2_buffer.field to ANY
0:00:00.187980649  1099 0xb3a03200 WARN            v4l2videoenc gstv4l2videoenc.c:670:gst_v4l2_video_enc_loop:<v4l2h264enc0> Encoder is producing too many buffers

After some time (seconds, to minutes) I get a critical error and the stream stops:

Code: Select all

** (gst-launch-1.0:1099): CRITICAL **: 10:17:10.641: gst_video_meta_map: assertion '!(flags & GST_MAP_WRITE) || gst_buffer_is_writable (meta->buffer)' failed
0:00:09.029859130  1099  0x13f9db0 ERROR                default video-frame.c:161:gst_video_frame_map_id: failed to map video frame plane 0
0:00:09.030365147  1099  0x13f9db0 ERROR         v4l2bufferpool gstv4l2bufferpool.c:161:gst_v4l2_buffer_pool_copy_buffer:<v4l2h264enc0:pool:sink> could not map buffer
0:00:09.030428313  1099  0x13f9db0 ERROR         v4l2bufferpool gstv4l2bufferpool.c:1975:gst_v4l2_buffer_pool_process:<v4l2h264enc0:pool:sink> failed to prepare data
0:00:09.031184200  1099  0x13f9db0 WARN            v4l2videoenc gstv4l2videoenc.c:803:gst_v4l2_video_enc_handle_frame:<v4l2h264enc0> error: Verarbeiten des Einzelbilds schlug fehl.
0:00:09.032068809  1099  0x13f9db0 WARN            v4l2videoenc gstv4l2videoenc.c:803:gst_v4l2_video_enc_handle_frame:<v4l2h264enc0> error: Maybe be due to not enough memory or failing driver
0:00:09.034387359  1099  0x13f9db0 WARN                 basesrc gstbasesrc.c:3055:gst_base_src_loop:<v4l2src0> error: Internal data stream error.
0:00:09.034676043  1099  0x13f9db0 WARN                 basesrc gstbasesrc.c:3055:gst_base_src_loop:<v4l2src0> error: streaming stopped, reason error (-5)
FEHLER: Von Element /GstPipeline:pipeline0/v4l2h264enc:v4l2h264enc0: Verarbeiten des Einzelbilds schlug fehl.
0:00:09.035074357  1099 0xb3a03200 WARN           v4l2allocator gstv4l2allocator.c:1349:gst_v4l2_allocator_dqbuf:<v4l2h264enc0:pool:src:allocator> V4L2 provided buffer has bytesused 0 which is too small to include data_offset 0
Zusätzliche Fehlerdiagnoseinformation:
gstv4l2videoenc.c(803): gst_v4l2_video_enc_handle_frame (): /GstPipeline:pipeline0/v4l2h264enc:v4l2h264enc0:
Maybe be due to not enough memory or failing driver
Execution ended after 0:00:08.932492101
Leitung wird auf PAUSIERT gesetzt ...
Leitung wird auf BEREIT gesetzt ...
0:00:09.039820456  1099  0x13fe5c0 WARN              bufferpool gstbufferpool.c:1394:gst_buffer_pool_set_flushing:<v4l2h264enc0:pool:sink> can't change flushing state of inactive pool
0:00:09.040366344  1099  0x13fe5c0 WARN              bufferpool gstbufferpool.c:1394:gst_buffer_pool_set_flushing:<v4l2h264enc0:pool:sink> can't change flushing state of inactive pool
0:00:09.040820231  1099  0x13fe5c0 WARN              bufferpool gstbufferpool.c:1394:gst_buffer_pool_set_flushing:<v4l2h264enc0:pool:sink> can't change flushing state of inactive pool
Leitung wird auf NULL gesetzt ...
Leitung wird geleert ...

I already tried to experiment with io modes according to other threads in the forum. But I was not able to get a working setup.

The video input is detected correctly and configured using an EDID file:

Code: Select all

   [ 4721.750020] unicam fe801000.csi: =================  START STATUS  =================
   [ 4721.751717] tc358743 0-000f: -----Chip status-----
   [ 4721.752340] tc358743 0-000f: Chip ID: 0x00
   [ 4721.752961] tc358743 0-000f: Chip revision: 0x00
   [ 4721.752973] tc358743 0-000f: Reset: IR: 1, CEC: 1, CSI TX: 0, HDMI: 0
   [ 4721.752983] tc358743 0-000f: Sleep mode: off
   [ 4721.752992] tc358743 0-000f: Cable detected (+5V power): yes
   [ 4721.753523] tc358743 0-000f: DDC lines enabled: yes
   [ 4721.754052] tc358743 0-000f: Hotplug enabled: yes
   [ 4721.754673] tc358743 0-000f: CEC enabled: no
   [ 4721.754682] tc358743 0-000f: -----Signal status-----
   [ 4721.754691] tc358743 0-000f: TMDS signal detected: yes
   [ 4721.754700] tc358743 0-000f: Stable sync signal: yes
   [ 4721.754709] tc358743 0-000f: PHY PLL locked: yes
   [ 4721.754718] tc358743 0-000f: PHY DE detected: yes
   [ 4721.765797] tc358743 0-000f: Detected format: 1280x720p50.0 (1980x750)
   [ 4721.765810] tc358743 0-000f: horizontal: fp = 0, -sync = 700, bp = 0
   [ 4721.765821] tc358743 0-000f: vertical: fp = 0, -sync = 30, bp = 0
   [ 4721.765831] tc358743 0-000f: pixelclock: 74250000
   [ 4721.765843] tc358743 0-000f: flags (0x0):
   [ 4721.765853] tc358743 0-000f: standards (0x0):
   [ 4721.765867] tc358743 0-000f: Configured format: 1280x720p50.0 (1980x750)
   [ 4721.765877] tc358743 0-000f: horizontal: fp = 0, -sync = 700, bp = 0
   [ 4721.765887] tc358743 0-000f: vertical: fp = 0, -sync = 30, bp = 0
   [ 4721.765897] tc358743 0-000f: pixelclock: 74250000
   [ 4721.765908] tc358743 0-000f: flags (0x0):
   [ 4721.765918] tc358743 0-000f: standards (0x0):
   [ 4721.765927] tc358743 0-000f: -----CSI-TX status-----
   [ 4721.765938] tc358743 0-000f: Lanes needed: 1
   [ 4721.765947] tc358743 0-000f: Lanes in use: 1
   [ 4721.766569] tc358743 0-000f: Waiting for particular sync signal: no
   [ 4721.767190] tc358743 0-000f: Transmit mode: no
   [ 4721.767809] tc358743 0-000f: Receive mode: no
   [ 4721.768429] tc358743 0-000f: Stopped: no
   [ 4721.768439] tc358743 0-000f: Color space: YCbCr 422 16-bit
   [ 4721.768968] tc358743 0-000f: -----HDMI status-----
   [ 4721.768977] tc358743 0-000f: HDCP encrypted content: no
   [ 4721.768987] tc358743 0-000f: Input color space: RGB limited range
   [ 4721.769517] tc358743 0-000f: AV Mute: off
   [ 4721.770046] tc358743 0-000f: Deep color mode: 8-bits per channel
   [ 4721.778810] tc358743 0-000f: HDMI infoframe: Auxiliary Video Information (AVI), version 2, length 13
   [ 4721.778823] tc358743 0-000f:     colorspace: RGB
   [ 4721.778835] tc358743 0-000f:     scan mode: No Data
   [ 4721.778847] tc358743 0-000f:     colorimetry: ITU709
   [ 4721.778858] tc358743 0-000f:     picture aspect: No Data
   [ 4721.778869] tc358743 0-000f:     active aspect: Same as Picture
   [ 4721.778880] tc358743 0-000f:     itc: IT Content
   [ 4721.778892] tc358743 0-000f:     extended colorimetry: xvYCC 709
   [ 4721.778903] tc358743 0-000f:     quantization range: Default
   [ 4721.778914] tc358743 0-000f:     nups: Unknown Non-uniform Scaling
   [ 4721.778925] tc358743 0-000f:     video code: 19
   [ 4721.778936] tc358743 0-000f:     ycc quantization range: Limited
   [ 4721.778947] tc358743 0-000f:     hdmi content type: Photo
   [ 4721.778958] tc358743 0-000f:     pixel repeat: 0
   [ 4721.778970] tc358743 0-000f:     bar top 0, bottom 0, left 0, right 0
   [ 4721.778981] unicam fe801000.csi: -----Receiver status-----
   [ 4721.778992] unicam fe801000.csi: V4L2 width/height:   1280x720
   [ 4721.779002] unicam fe801000.csi: Mediabus format:     0000200f
   [ 4721.779013] unicam fe801000.csi: V4L2 format:         UYVY
   [ 4721.779023] unicam fe801000.csi: Unpacking/packing:   0 / 0
   [ 4721.779031] unicam fe801000.csi: ----Live data----
   [ 4721.779041] unicam fe801000.csi: Programmed stride:   2560
   [ 4721.779051] unicam fe801000.csi: Detected resolution: 0x0
   [ 4721.779061] unicam fe801000.csi: Write pointer:       e9500000
   [ 4721.779071] unicam fe801000.csi: ==================  END STATUS  ==================

I also monitored cpu usage, mem and temperature and nothing seems to overload the Pi.

I am stuck with this problem for days and do not know what to do next. Any help appreciated!

[NKnuelle]

Update:
I recently tested the pipeline using io-modes 2 on Ouput AND v4l2src like this (something I missed experimenting before):

Code: Select all

GST_DEBUG=2 gst-launch-1.0 flvmux streamable=true name=mux ! rtmpsink location="rtmp://localhost:1935/live/stream live=1" v4l2src io-mode=2 ! "video/x-raw,framerate=50/1,format=UYVY" ! v4l2h264enc output-io-mode=2 extra-controls="controls,video_bitrate=6000000;" ! video/x-h264,profile=high ! h264parse ! queue ! mux.

I am still getting the warnings but the stream is stable for about 2 hours now. The CPU usage is higher, because as far as I understand, mode 2 means memory buffering which involves the CPU for copying frames. The overall CPU usage on all 4 cores is about 30-40%.

This is still just a workaround and using DMA buffers would reduce CPU usage and would be preferable over memory buffering.

[6by9]

Sorry, I don't have a tc358743 system set up at present to experiment with. I do need to investigate something on it in the next few days, so will try to find a few moments to look into this. The fact that it takes a fair while to go wrong counts against it.

mode 2 does mean memcpying the buffers around, which isn't ideal.

Please confirm the kernel and firmware versions you are running. "uname -a" and "vcgencmd version".

[NKnuelle]

Hey 6by9!

thank you for your response.

The Kernel and Firmware versions are:

Code: Select all

pi@raspberrypi:~ $ uname -a
Linux raspberrypi 4.19.97-v7l+ #1294 SMP Thu Jan 30 13:21:14 GMT 2020 armv7l GNU/Linux
pi@raspberrypi:~ $ vcgencmd version
Feb 12 2020 12:37:37 
Copyright (c) 2012 Broadcom
version c3c8dbdf147686fb0c3f32aece709d0653368810 (clean) (release) (start_x)

[NKnuelle]

I am still experiencing the buffer issues with io-modes 4/5. Is there any solution yet? Should I open up an issue on Github for the GStreamer Plugin or does it relate to the Raspberry Pi/Auvidea module only?

[6by9]

Open an issue on https://github.com/raspberrypi/linux/issues so it doesn't get lost within the forums.
At present I couldn't say whether it was kernel or GStreamer causing the issue.

One situation that V4L2 doesn't handle well is what to do should the encoded frame be larger than the provided buffer. This can sometimes happen with realtime single-pass encoders where the predicted quantisation parameters end up being off. Typically a scene change the frame after an I-frame would do it, as the diffs are huge compared to the expected encode.
OpenMax and MMAL allow frames to be denoted as not being the end of frame, and then the next buffer will generally complete the frame. V4L2 doesn't, so we allocate large buffers to generally avoid issues (512kB/buffer for <=720P, and 768kB for >720P).
There were discussions at ELCE last year which I think concluded that the codec needs to throw an error and stop encoding under these conditions. It's not a great behaviour, but at least it is defined.

In fact my comment in the source code also shows that it is a problem with some pre-encoded streams

Code: Select all

/*
 * The unanswered question - what is the maximum size of a compressed frame?
 * V4L2 mandates that the encoded frame must fit in a single buffer. Sizing
 * that buffer is a compromise between wasting memory and risking not fitting.
 * The 1080P version of Big Buck Bunny has some frames that exceed 512kB.
 * Adopt a moderately arbitrary split at 720P for switching between 512 and
 * 768kB buffers.
 */
#define DEF_COMP_BUF_SIZE_GREATER_720P	(768 << 10)
#define DEF_COMP_BUF_SIZE_720P_OR_LESS	(512 << 10)

So it's not just encode that has issues.
I'm not aware of any way to increase the buffer size from the command line. You could try increasing those defines and recompiling the kernel (you don't say what resolution you're encoding), or dropping your bitrate.

Edit: Doh, your debug includes the information that it's 720p50.

[NKnuelle]

Thanks for your quick response. I will open up an issue.

I am trying to encode 720p50 which should be no problem for the card and the Pi. Because of the issues I didn't try to encode higher resolutions.

I already had issues using io-mode 4/5 using my camera which was filming a somehow static image (no big changes, no pan/tilt). So I think the issue is not only related to frame changes.

I also tried it with realtime TV program which has a lot of frame changes, but I didn't recognize a more critical behaviour.. The streaming mostly ended after ~5-10minutes with io-mode 4/5.

[NKnuelle]

Hi there,

Is there any update on this topic or anything else I could try? Maybe using something else than V4L2? The problem using io-mode 2 seems to be that the latency for RTMP is much higher (around 5-10seconds).

Greetings!

[6by9]

Sorry, it'd slipped off the list. Juggling tasks is a nightmare.

I need to do a couple of fixups to the tc358743 on the 5.4 branch for 4 lane mode, so I'll have an experiment.

[6by9]

There was a thread about V4L2 decode at https://github.com/raspberrypi/linux/issues/3560 that stated that GStreamer has altered the way it handles V4L2 buffers as of 1.16.

If you can rebuild GStreamer at a more recent version, then it's probably worth doing. Alternatively for a test, Gentoo generally uses the bleeding edge of almost all packages, so trying the Gentoo image may do that for you viewtopic.php?f=54&t=188448

[NKnuelle]

I managed to get GStreamer 1.16.2 up and working:

Code: Select all

gst-launch-1.0 --version
gst-launch-1.0 version 1.16.2
GStreamer 1.16.2
Unknown package origin

For this I followed the instructions in this repo:

Code: Select all

https://github.com/PietroAvolio/Building-Gstreamer-Raspberry-Pi-With-SRT-Support/blob/master/gstreamer-build.sh

Sadly the stream is again stopping after ~15m and I can't get it working again:

Code: Select all

GST_DEBUG=2 gst-launch-1.0 flvmux streamable=true name=mux ! rtmpsink location="rtmp://localhost:1935/live/stream live=1" v4l2src io-mode=4 ! "video/x-raw,framerate=50/1,format=UYVY" ! v4l2h264enc output-io-mode=5 extra-controls="controls,video_bitrate=6000000;" ! video/x-h264,profile=high ! h264parse ! queue ! mux.
0:00:00.004398980  5749   0xffe4c0 WARN         GST_PERFORMANCE gstbuffer.c:486:_priv_gst_buffer_initialize: No 64-bit atomic int defined for this platform/toolchain!
Leitung wird auf PAUSIERT gesetzt ...
0:00:00.129041237  5749   0xffe4c0 WARN                    v4l2 gstv4l2object.c:4211:gst_v4l2_object_probe_caps:<v4l2h264enc0:src> Failed to probe pixel aspect ratio with VIDIOC_CROPCAP: Das Argument ist ungültig
0:00:00.129986284  5749  0x111b490 WARN                 v4l2src gstv4l2src.c:696:gst_v4l2src_query:<v4l2src0> Can't give latency since framerate isn't fixated !
0:00:00.130056543  5749  0x111b490 WARN              aggregator gstaggregator.c:1757:gst_aggregator_query_latency_unlocked:<mux> Latency query failed
Leitung ist aktiv und erfordert keinen VORLAUF …
Leitung wird auf ABSPIELEN gesetzt ...
New clock: GstSystemClock
0:00:00.133145311  5749  0x111b490 WARN                 v4l2src gstv4l2src.c:696:gst_v4l2src_query:<v4l2src0> Can't give latency since framerate isn't fixated !
0:00:00.133184644  5749  0x111b490 WARN              aggregator gstaggregator.c:1757:gst_aggregator_query_latency_unlocked:<mux> Latency query failed
0:00:00.133503456  5749  0x111b490 WARN                 v4l2src gstv4l2src.c:696:gst_v4l2src_query:<v4l2src0> Can't give latency since framerate isn't fixated !
0:00:00.133535345  5749  0x111b490 WARN              aggregator gstaggregator.c:1757:gst_aggregator_query_latency_unlocked:<mux> Latency query failed
Verzögerung neu verteilen …
0:00:00.170260758  5749  0x11107b0 WARN          v4l2bufferpool gstv4l2bufferpool.c:810:gst_v4l2_buffer_pool_start:<v4l2h264enc0:pool:src> Uncertain or not enough buffers, enabling copy threshold

It seems like updating GStreamer didn't do the trick...

Also for your attention:
I tried using the SRT-Protocol which is now part of GStreamer but this gives me:

Code: Select all

GST_DEBUG=2 gst-launch-1.0 v4l2src ! video/x-raw, height=720, width=1280 ! videoconvert ! v4l2h264enc ! video/x-h264, profile=high ! mpegtsmux a
lignment=7 ! srtserversink uri=srt://0.0.0.0:8888/
0:00:00.004351351  5763  0x11884c0 WARN         GST_PERFORMANCE gstbuffer.c:486:_priv_gst_buffer_initialize: No 64-bit atomic int defined for this platform/toolchain!
Leitung wird auf PAUSIERT gesetzt ...
0:00:00.148277038  5763  0x11884c0 WARN                 srtsink gstsrtsink.c:160:gst_srt_sink_start:<srtsink0> error: Failed to open SRT: failed to set SRTO_LINGER (reason: Operation not supported: Bad parameters)
0:00:00.148401574  5763  0x11884c0 WARN                basesink gstbasesink.c:5368:gst_base_sink_change_state:<srtsink0> error: Failed to start
FEHLER: Leitung möchte nicht pausiert werden.
FEHLER: Von Element /GstPipeline:pipeline0/GstSRTSink:srtsink0: Die Ressource konnte nicht zum Schreiben geöffnet werden.
Zusätzliche Fehlerdiagnoseinformation:
gstsrtsink.c(160): gst_srt_sink_start (): /GstPipeline:pipeline0/GstSRTSink:srtsink0:
Failed to open SRT: failed to set SRTO_LINGER (reason: Operation not supported: Bad parameters)
Leitung wird auf NULL gesetzt ...
Leitung wird geleert ...

Some of the error logs are in german, if you need some additional information I can translate it if necessary.