Dropped packets over B+ Ethernet (USB issue?)

[Malvineous]

Hi all,

I'm trying to debug a problem that is causing video corruption when streaming the Pi's video over the network. Recording locally works fine, I only see video corruption when streaming, so it looks like a problem with the NIC either dropping packets or taking too long to send them.

I am using this command to send the stream over the network:

Code: Select all

v4l2-ctl -v width=1296,height=972,pixelformat=H264 \
        --set-ctrl=exposure_dynamic_framerate=1 \
        --set-ctrl=video_bitrate=3000000 \
        --set-ctrl=scene_mode=8

ffmpeg -f h264 -probesize 32 -r 30 -i /dev/video0 -vcodec copy -an -f rtp_mpegts udp://224.0.1.2:5004 -loglevel fatal

On the receiving side I am using this to view the video:

Code: Select all

ffplay -sync ext -an -fast -framedrop -probesize 32 udp://224.0.1.2:5004

The higher the bitrate the worse the problem, and at 10Mbps it's pretty much unwatchable.

Has anyone else seen this type of behaviour? Is there anything I can do to tweak the B+ NIC or USB configuration that might result in fewer dropped packets?

I'm seeing this same issue on both the build in Ethernet and a USB wifi stick, so it's possible it's more a USB problem rather than Ethernet.

Any suggestions on where I can find details about what options there are as far as tweaking the USB/Ethernet controller?

Many thanks!

[allfox]

I don't know about camera.

I used to have an original model B Pi, it's a slow machine, can not fill up it's own USB bus when copy file via on board ethernet.

But you are talking about 10MBits. I think even original Pi are slow, they could handle 10MBits transfering.

I looked your settings, while I never used camera. The ffmpeg just copying things into network, it looks like hardware AVC encoding to me.

The behavior of dropping packet during network is normal. And in fact, "cleverly drop packet if taking too long to send" is an advanced routing technic named CoDel. It should improve streaming performance, rather than hurt. You don't get it by default.

I would suggest:
1 Check the Pi spec to tell how fast VC4 could encode AVC.
2 When streaming, check the CPU usage rate. Maybe B+ is not fast enough to copy and pack up those frames.
3 As I know nothing about camera, so I can't tell if those parameters are suitable.

[allfox]

It suddely comes to me that there is a tool named ncat. It directly write data from local into network. So I'm thinking about directly send /dev/video0 into network, without ffmpeg. Maybe faster.

[Malvineous]

Thanks for your reply!

I don't think it's a CPU problem as ffmpeg uses less than 10% CPU. I don't think ncat would work because ffmpeg adds various headers onto the data to make it easier to play at the remote end, and it seems troublesome to play the raw H264 data obtained directly from the GPU. Also, ffmpeg handles multicast RTP - which is important because I receive three copies of the stream, and with multicast the Pi only needs to upload the data once. It really might max out the Pi's bandwidth if it had to upload the same data to three different unicast IP addresses! It's worth noting that the video corruption is identical across the three different streams, which suggests the packets that go missing were never sent. (If they were sent but lost in transit, you'd expect the packets to arrive on some PCs but not others, resulting in a different corruption pattern. But the corruption pattern is identical across multiple receiving PCs.) I haven't found any multicast-supporting alternative I can easily test with apart from ffmpeg unfortunately. I would imagine if I switched from UDP to TCP then the corruption would go away, but it is not possible to multicast TCP.

I don't think it's a camera/codec problem as I can record from the camera to an SD card at 5Mbps, but I get video corruption when streaming at 2Mbps. If the hardware was underpowered you would expect the SD card recording to also have at least some corruption, but there is none.

I think you might be right in that packets are being dropped if they take too long to transmit. Unfortunately with H264 it does not handle lost packets gracefully so it is very noticeable.

I am not sure how to further investigate if or why packets are being dropped, so I'm open to any suggestions!

[jamesh]

Any messages that look relevant in `dmesg`?

[Malvineous]

Nope, nothing in dmesg, the logs, even the ifconfig counters aren't showing any dropped packets so it's rather confusing.

I tried lowering the UDP multicast packet size to 200 bytes and this seemed to reduce the video corruption, but only at 30fps. Once night falls and the camera drops below around 5fps, then the 200 byte packet size makes the problem much worse.

Making the packets larger than 1500 bytes causes fragmentation (according to tcpdump on the receiving PC) but it doesn't seem to affect the video quality (still the usual level of picture corruption).

[Malvineous]

I investigated using the "iperf" performance monitoring tool and got some interesting results.

Turns out that the Pi can transmit TCP data at 40Mbps, and it can transmit UDP at 65Mbps. Yet it still manages to drop a few packets regardless of the speed.

Here's what I used to run the test:

Code: Select all

pc$ iperf -s                       # Start TCP server on PC
pi$ iperf -c ip-of-pc              # Send TCP traffic to PC as fast as possible

pc$ iperf -u -s                    # Start UDP server on PC
pi$ iperf -u -c ip-of-pc -b 100M   # Send UDP traffic to PC as close to 100Mbps as possible

This is what the server said for the TCP tests:

Code: Select all

[ ID] Interval       Transfer     Bandwidth
[  4]  0.0-10.0 sec  47.1 MBytes  39.4 Mbits/sec

And for UDP, there is some loss too. The first two tests were at the default iperf UDP limit of 1Mbps, the second I raised the limit to 100Mbps with "-b 100M" on the Pi.

Code: Select all

[ ID] Interval       Transfer     Bandwidth        Jitter   Lost/Total Datagrams
[  3]  0.0-10.0 sec  1.25 MBytes  1.05 Mbits/sec   0.043 ms    3/  893 (0.34%)
[  4]  0.0-10.0 sec  1.25 MBytes  1.05 Mbits/sec   0.052 ms    2/  893 (0.22%)

[  3]  0.0-10.0 sec  77.7 MBytes  65.2 Mbits/sec   0.020 ms    2/55419 (0.0036%)

[allfox]

I ordered a camera, it delivered today.

I'm getting a quite watchable video via UDP. I only have one screen, so can't be sure the multi-cast performance, could you try it?

Sending site is a Raspberry Pi A+, you couldn't have a Pi slower than this.
Receiving site is a laptop running Ubuntu 16.04 LTS.

Sending Pi is enabled fq_codel queuing discipline. You could do the same by create a /etc/dhcpcd.exit-hook:

Code: Select all

#!/bin/sh

ETHTOOL="/sbin/ethtool"
TC="/sbin/tc"

$ETHTOOL --offload "$interface" gro off
$ETHTOOL --offload "$interface" lro off
$ETHTOOL --offload "$interface" gso off
$ETHTOOL --offload "$interface" tso off
$ETHTOOL --offload "$interface" ufo off

echo -n "$interface" | grep -E '^eth[[:digit:]]+$'
check=$?
if [ "$check" -eq 0 ]
then
        $TC qdisc del dev "$interface" root
        $TC qdisc add dev "$interface" root fq_codel noecn
fi

exit 0

Sending command: raspivid --inline -t 0 --output - | avconv -re -f h264 -i - -vcodec copy -tune zerolatency -an -f mpegts udp://224.0.0.2:61234

Receiving command: ffplay -i udp://224.0.0.2:61234

I don't know those streaming protocol, have no idea what this "mpegts" is. I guess I would check them later. It's 5:36 AM.