rma153
Posts: 53
Joined: Sat Nov 12, 2016 6:17 am

Random Greyed-out frames in gstreamer video stream

Sun Apr 11, 2021 9:46 pm

I am seeing annoying random greyed-out frames

I am successfully streaming my GoPro (1080P/30fps) video using RPi4B with Auvidea B101 HDMI to CSI2 board over Zero Tier VPN over 5G cellular link to my remote windows 10 Mission Planner Groundstation laptop.

The working pipeline on my RPi4 is below (RPi4 is at KERNEL 5.10, gstreamer 1.18.4)

Code: Select all

gst-launch-1.0 v4l2src device=/dev/video0 ! 'video/x-raw,framerate=30/1,format=UYVY' ! v4l2h264enc ! 'video/x-h264,level=(string)4' ! rtph264pay ! udpsink host=172.30.xxx.xxx port=5600
Note: No greyed-out frames appear when I display the stream directly to my local RPi HDMI monitor, only when streamed to my remote windows laptop)

What can I modify in the two pipelines to fix this?

The pipeline on my remote windows 10 machine is below. This works (very low latency), but I am getting annoying random greyed-out frames that generally last at most around .25 seconds at various times in the stream. Note this doesn't happen if I use a Pi-camera on the same set-up.

Code: Select all

gst-launch-1.0 -v udpsrc port=5600 caps="application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)H264" ! rtpjitterbuffer ! rtph264depay ! avdec_h264 ! videoconvert ! autovideosink sync=false

rma153
Posts: 53
Joined: Sat Nov 12, 2016 6:17 am

Re: Random Greyed-out frames in gstreamer video stream

Sun Apr 11, 2021 11:25 pm

[img][
image (1).png
image (1).png (191.22 KiB) Viewed 391 times
/img]

To further describe the issue, the greyed-out Image attached shows that when the stream reverts momentarily to an all greyed-out image, it typically will only partially display colored objects that are in motion, while the static objects in the surrounding background all display as greyed out (perhaps related to the I-frames?)

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 11018
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Random Greyed-out frames in gstreamer video stream

Mon Apr 12, 2021 11:05 am

That looks like part of the stream has been dropped, and therefore the decoder is working with an invalid reference frame. How the decoder works in these situations is implementation dependent.

What bitrate are you encoding at, and can the 5G link sustain that reliably?
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

rma153
Posts: 53
Joined: Sat Nov 12, 2016 6:17 am

Re: Random Greyed-out frames in gstreamer video stream

Mon Apr 12, 2021 11:46 pm

Here's what I see om my RPi4 (what command should I use to show the Bitrate)

Status Log:

[150156.988386] unicam fe801000.csi: ================= START STATUS =================
[150156.991192] tc358743 10-000f: -----Chip status-----
[150156.991823] tc358743 10-000f: Chip ID: 0x00
[150156.992449] tc358743 10-000f: Chip revision: 0x00
[150156.992463] tc358743 10-000f: Reset: IR: 1, CEC: 1, CSI TX: 0, HDMI: 0
[150156.992475] tc358743 10-000f: Sleep mode: off
[150156.992487] tc358743 10-000f: Cable detected (+5V power): yes
[150156.993032] tc358743 10-000f: DDC lines enabled: yes
[150156.993569] tc358743 10-000f: Hotplug enabled: yes
[150156.994195] tc358743 10-000f: CEC enabled: no
[150156.994206] tc358743 10-000f: -----Signal status-----
[150156.994218] tc358743 10-000f: TMDS signal detected: yes
[150156.994229] tc358743 10-000f: Stable sync signal: yes
[150156.994240] tc358743 10-000f: PHY PLL locked: yes
[150156.994251] tc358743 10-000f: PHY DE detected: yes
[150157.001227] tc358743 10-000f: Detected format: 1920x1080p30.00 (2200x1125)
[150157.001242] tc358743 10-000f: horizontal: fp = 0, -sync = 280, bp = 0
[150157.001255] tc358743 10-000f: vertical: fp = 0, -sync = 45, bp = 0
[150157.001266] tc358743 10-000f: pixelclock: 74250000
[150157.001280] tc358743 10-000f: flags (0x0):
[150157.001292] tc358743 10-000f: standards (0x0):
[150157.001306] tc358743 10-000f: Configured format: 1920x1080p30.00 (2200x1125)
[150157.001319] tc358743 10-000f: horizontal: fp = 0, -sync = 280, bp = 0
[150157.001331] tc358743 10-000f: vertical: fp = 0, -sync = 45, bp = 0
[150157.001342] tc358743 10-000f: pixelclock: 74250000
[150157.001355] tc358743 10-000f: flags (0x0):
[150157.001366] tc358743 10-000f: standards (0x0):
[150157.001377] tc358743 10-000f: -----CSI-TX status-----
[150157.001389] tc358743 10-000f: Lanes needed: 2
[150157.001400] tc358743 10-000f: Lanes in use: 2
[150157.002039] tc358743 10-000f: Waiting for particular sync signal: no
[150157.002676] tc358743 10-000f: Transmit mode: no
[150157.003306] tc358743 10-000f: Receive mode: no
[150157.003931] tc358743 10-000f: Stopped: no
[150157.003942] tc358743 10-000f: Color space: YCbCr 422 16-bit
[150157.004478] tc358743 10-000f: -----HDMI status-----
[150157.004492] tc358743 10-000f: HDCP encrypted content: no
[150157.004506] tc358743 10-000f: Input color space: RGB limited range
[150157.005049] tc358743 10-000f: AV Mute: off
[150157.005592] tc358743 10-000f: Deep color mode: 8-bits per channel
[150157.008264] tc358743 10-000f: HDMI infoframe: Auxiliary Video Information (AVI), version 2, length 13
[150157.008291] tc358743 10-000f: colorspace: RGB
[150157.008304] tc358743 10-000f: scan mode: Underscan
[150157.008316] tc358743 10-000f: colorimetry: ITU601
[150157.008329] tc358743 10-000f: picture aspect: 16:9
[150157.008344] tc358743 10-000f: active aspect: 14:9
[150157.008356] tc358743 10-000f: itc: No Data
[150157.008371] tc358743 10-000f: extended colorimetry: xvYCC 601
[150157.008384] tc358743 10-000f: quantization range: Limited
[150157.008396] tc358743 10-000f: nups: Unknown Non-uniform Scaling
[150157.008409] tc358743 10-000f: video code: 34
[150157.008422] tc358743 10-000f: ycc quantization range: Limited
[150157.008434] tc358743 10-000f: hdmi content type: Graphics
[150157.008447] tc358743 10-000f: pixel repeat: 0
[150157.008463] tc358743 10-000f: bar top 0, bottom 0, left 0, right 0
[150157.008478] unicam fe801000.csi: -----Receiver status-----
[150157.008492] unicam fe801000.csi: V4L2 width/height: 1920x1080
[150157.008505] unicam fe801000.csi: Mediabus format: 0000200f
[150157.008517] unicam fe801000.csi: V4L2 format: 59565955
[150157.008530] unicam fe801000.csi: Unpacking/packing: 0 / 0
[150157.008540] unicam fe801000.csi: ----Live data----
[150157.008552] unicam fe801000.csi: Programmed stride: 3840
[150157.008564] unicam fe801000.csi: Detected resolution: 1920x1080
[150157.008576] unicam fe801000.csi: Write pointer: e05f2f10
[150157.008588] unicam fe801000.csi: ================== END STATUS ==================
pi@raspberrypi:~ $ v4l2-ctl --query-dv-timings
Active width: 1920
Active height: 1080
Total width: 2200
Total height: 1125
Frame format: progressive
Polarities: -vsync -hsync
Pixelclock: 74250000 Hz (30.00 frames per second)
Horizontal frontporch: 0
Horizontal sync: 280
Horizontal backporch: 0
Vertical frontporch: 0
Vertical sync: 45
Vertical backporch: 0
Standards:
Flags:
pi@raspberrypi:~ $ v4l2-ctl --set-dv-bt-timings query
BT timings set
pi@raspberrypi:~ $ v4l2-ctl -V
Format Video Capture:
Width/Height : 1920/1080
Pixel Format : 'UYVY' (UYVY 4:2:2)
Field : None
Bytes per Line : 3840
Size Image : 4147200
Colorspace : SMPTE 170M
Transfer Function : Default (maps to Rec. 709)
YCbCr/HSV Encoding: Default (maps to ITU-R 601)
Quantization : Default (maps to Limited Range)
Flags :
pi@raspberrypi:~ $

rma153
Posts: 53
Joined: Sat Nov 12, 2016 6:17 am

Re: Random Greyed-out frames in gstreamer video stream

Tue Apr 13, 2021 12:29 am

My 5G speedtest on the RPI4 sender shows on average 22Mbps DS and 9Mbps US. My 5G speedtest on the receiving end (Windows 10, i7) is 30Mbps DS and 10Mbps US. Connected via Zero Tier VPN.

rma153
Posts: 53
Joined: Sat Nov 12, 2016 6:17 am

Re: Random Greyed-out frames in gstreamer video stream

Tue Apr 13, 2021 3:00 am

Looks like my 5G upstream link from my RPi4 is less than the default v4l2 codec video_bitrate of 10Mbps shown below. How do I correctly change the default bitrate to be below my 5G bitrate, in the pipeline? This should fix the issue of Greyed-out frames, correct?

Code: Select all

pi@raspberrypi:~ $ v4l2-ctl -d 11 --list-ctrls-menu

Codec Controls

             video_bitrate_mode 0x009909ce (menu)   : min=0 max=1 default=0 value=0 flags=update
				0: Variable Bitrate
				1: Constant Bitrate
                  video_bitrate 0x009909cf (int)    : min=25000 max=25000000 step=25000 default=10000000 value=10000000
         repeat_sequence_header 0x009909e2 (bool)   : default=0 value=0
            h264_i_frame_period 0x00990a66 (int)    : min=0 max=2147483647 step=1 default=60 value=60
                     h264_level 0x00990a67 (menu)   : min=0 max=13 default=11 value=11
				0: 1
				1: 1b
				2: 1.1
				3: 1.2
				4: 1.3
				5: 2
				6: 2.1
				7: 2.2
				8: 3
				9: 3.1
				10: 3.2
				11: 4
				12: 4.1
				13: 4.2
                   h264_profile 0x00990a6b (menu)   : min=0 max=4 default=4 value=4
				0: Baseline
				1: Constrained Baseline
				2: Main
				4: High

rma153
Posts: 53
Joined: Sat Nov 12, 2016 6:17 am

Re: Random Greyed-out frames in gstreamer video stream

Tue Apr 13, 2021 3:55 am

Or, instead of a pipeline modification, should I use the following command to change the v4l2 bitrate

Code: Select all

v4l2-ctl --set-ctrl video_bitrate=7000000

rma153
Posts: 53
Joined: Sat Nov 12, 2016 6:17 am

Re: Random Greyed-out frames in gstreamer video stream

Tue Apr 13, 2021 4:55 am

Well, that doesn't work so next I'll try adding this to my pipeline

Code: Select all

v4l2h264enc extra-controls="foo,video_bitrate=7000000"

rma153
Posts: 53
Joined: Sat Nov 12, 2016 6:17 am

Re: Random Greyed-out frames in gstreamer video stream

Tue Apr 13, 2021 5:42 am

Code: Select all

v4l2h264enc extra-controls="foo,video_bitrate=6000000"
Adding this to the pipeline helps (dramatically shortens the duration of Grey frames), but doesn't eliminate them entirely. Are teher any other extra controls I should try?

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 11018
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Random Greyed-out frames in gstreamer video stream

Tue Apr 13, 2021 12:15 pm

Bitrate is the main parameter, so you've sussed that one.

The bitrate is the target - it will vary slightly above/below as the quantisation parameters are set before the frame is encoded, and then the control algorithm will adjust for the next frame should it over or under shoot the target. I frames obviously get more bits assigned to them than P-frames.
If you run too close to the limit of your link, then any overshoot may result in a moderate amount of time to restore equilibrium.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

rma153
Posts: 53
Joined: Sat Nov 12, 2016 6:17 am

Re: Random Greyed-out frames in gstreamer video stream

Tue Apr 13, 2021 7:43 pm

ok. Thank you 6x9. This helped. The video is stunning and low latency. I'm now convinced that while running through the Zero Tier VPN to Tmobile the link is hitting multiple NAT layers, causing Zero Tier to struggle punching through and causing the glitch. I'll try OpenVpn next. Hoping that will solve the occasional glitch.

rma153
Posts: 53
Joined: Sat Nov 12, 2016 6:17 am

Re: Random Greyed-out frames in gstreamer video stream

Thu Apr 22, 2021 1:09 am

6x9

Perhaps this would help eliminate the greyed -out frames issue

V4L2 driver - option to specify the intra refresh period (key frame rate/GoP)

h264_i_frame_period (int) : min=1 max=1000 step=1 default=60 value=60

is this control supported?
v4l2-ctl --set-ctrl=h264_i_frame_period=1

But when I try it, I get

unknown control 'h264_i_frame_period'

rma153
Posts: 53
Joined: Sat Nov 12, 2016 6:17 am

Re: Random Greyed-out frames in gstreamer video stream

Thu Apr 22, 2021 11:42 pm

so, i added
h264_i_frame_period=2
to the pipeline, and this helped to shorten the grey-outs to no more than a blink, but they still occur very frequently. I also tried all the different GoPro resolutions available, to no avail.

Are there any other attributes I can add to the sender pipeline that I can try that might fix this annoying behavior? Perhaps set/increase rtpjitterbuffer in the receiver pipeline?

Return to “Camera board”