Page 1 of 1

Packet loss/dropout with iperf3 - UDP

Posted: Fri Mar 22, 2019 7:01 pm
by mkirk
iperf3 tests are run with UDP between 2 Pi 3B boards over the on-board WiFi ( channel 6 = 2.437 GHz)
Server Pi is set up as Access Point (hostapd/dnsmasq) and client Pi is attached.

Below is an example where iperf3 has zero data transfer for a full 1 second period (32-33 seconds).
Just prior there is high jitter with packet loss and after the event as well.

Setup is running in a RF screen room with no interfering WiFi devices.
RF spectrum analyzer (2.437 GHz) shows loss of WiFi RF signal during this event - so Pi is not transmitting and nothing else is interfering.
Bluetooth is disabled on both Pi's.

Code: Select all

 iperf3 -c 192.168.0.1 -t 60 -f m -u -b 50M 
      <<<  *******   SKIP TO MIDPOINT OF IPERF3 TEST ************* >>>>
[  5]  26.00-27.00  sec  5.20 MBytes  43.6 Mbits/sec  1.020 ms  0/666 (0%)  
[  5]  27.00-28.00  sec  5.02 MBytes  42.1 Mbits/sec  1.105 ms  0/643 (0%)  
[  5]  28.00-29.00  sec  2.88 MBytes  24.1 Mbits/sec  0.955 ms  0/368 (0%)      <<< WHY FEWER DATAGRAMS HERE?
[  5]  29.00-30.00  sec  5.13 MBytes  43.1 Mbits/sec  1.166 ms  0/657 (0%)  
[  5]  30.00-31.00  sec  5.23 MBytes  43.8 Mbits/sec  0.944 ms  0/669 (0%)  
[  5]  31.00-32.00  sec   144 KBytes  1.18 Mbits/sec  20.133 ms  26/44 (59%)    <<<  HIGH JITTER, PACKET LOSS
[  5]  32.00-33.00  sec  0.00 Bytes  0.00 Mbits/sec  20.133 ms  0/0 (0%)     <<< COMPLETE LOSS - NO DATA TRANSFER
[  5]  33.00-34.00  sec   136 KBytes  1.12 Mbits/sec  156.233 ms  11/28 (39%)    <<< RECOVERY, HIGH JITTER, LOW PACKET TRANSFER
[  5]  34.00-35.00  sec  2.15 MBytes  18.0 Mbits/sec  1.175 ms  0/275 (0%)      << BACK TO NORMAL
[  5]  35.00-36.00  sec  5.16 MBytes  43.3 Mbits/sec  0.981 ms  0/660 (0%)  
[  5]  36.00-37.00  sec  2.78 MBytes  23.3 Mbits/sec  6.918 ms  1/357 (0.28%)  
[  5]  37.00-38.00  sec  5.00 MBytes  41.9 Mbits/sec  1.020 ms  0/640 (0%)  
[  5]  38.00-39.00  sec  5.23 MBytes  43.9 Mbits/sec  1.090 ms  0/670 (0%)  
[  5]  39.00-40.00  sec  5.24 MBytes  44.0 Mbits/sec  1.171 ms  0/671 (0%)  
<<<< ************ CUT HERE  TO END OF TEST  ************* >>>> 
[  5]  59.00-60.00  sec  2.67 MBytes  22.4 Mbits/sec  6.552 ms  0/342 (0%)  <<<<<< END OF TEST
[  5]  60.00-60.07  sec   384 KBytes  43.7 Mbits/sec  1.505 ms  0/48 (0%)  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Jitter    Lost/Total Datagrams
[  5]   0.00-60.07  sec  0.00 Bytes  0.00 Mbits/sec  1.505 ms  39/34519 (0.11%)   <<  THE 32-33s INTERVAL ABOVE IS NOT INCLUDED HERE.
Can some parameter be changed in iperf3 (buffer sizing?) or in the WiFi configuration to avoid loss?
Or is there a known issue in the WiFi chipset or driver that could cause this loss?

It appears that iperf3 just hangs during this event - the stats do not reflect the gap, only the lost datagrams before and after.

Similar event is seen with WiFi TCP test - but only with Pi AP as client and attached Pi as server.
Reversing roles only has a few lower datagram rate events (loss), but never complete dropout.

Running the same test with a wired 100Mbps Ethernet connection yields 0% datagram loss (rate is 93.2 Mbps).

Any insights appreciated.

Re: Packet loss/dropout with iperf3 - UDP

Posted: Fri Mar 22, 2019 8:17 pm
by mkirk
Some progress on this issue.

For TCP tests, iperf3 bandwidth is unlimited which means it will try and push as much data as possible through the link.
This would cause the task to run out of buffers and require a buffer recovery. During this interval, no data is sent.
It would not be caused by the buffer size or packet sizes used.

For UDP test, the default bandwidth is 1 Mbps. Running at 50 Mbps caused the same issue as seen on TCP.

Running iperf3 TCP and UDP with restricted bandwidth (25 Mbps) over WiFi seems to fix the problem.
More tests are planned to confirm.

Re: Packet loss/dropout with iperf3 - UDP

Posted: Fri Apr 05, 2019 4:49 pm
by mkirk
New issue with this evaluation. I am getting a periodic (1 minute) dropout of WiFi signal (no RF on spectrum analyzer) using the nuttcp tool. I can't understand what would shutdown WiFi, the onboard Bluetooth is disabled. Log file is below.
One Pi is set up as Access Point (hostapd/dnsmasq) and other Pi is attached.

Attached Pi is the nuttcp server and AP Pi is the client.
Have tried 3 different Pi's as server and as client. Same result.
If I use my laptop as client, same result. - periodic 60 second WiFi dropout.

Does WiFi have to stop and do periodic calibration? Could WiFi be going in to sleep mode?

If i reverse roles, AP as server and attached Pi as client, I do not see these dropouts. There are cases where data rate drops down to 10-12 Mbps, but never zero.

CLIENT COMMAND: sudo nice -n -20 nuttcp -tvv -i 1 -frunningtotal -T240 -R20m 192.168.0.74
SERVER COMMAND: sudo nice -n -20 nuttcp -Sv

Code: Select all

2.3750 MB /   1.00 sec =   19.9230 Mbps     0 retrans Tot:    76.2500 MB /  32.00 sec =   19.9885 Mbps     0 retrans
    0.6250 MB /   1.00 sec =    5.2428 Mbps     0 retrans Tot:    76.8750 MB /  33.00 sec =   19.5416 Mbps     0 retrans
    0.0000 MB /   1.00 sec =    0.0000 Mbps     0 retrans Tot:    76.8750 MB /  34.00 sec =   18.9669 Mbps     0 retrans   
                       ***************   First loss – 34 seconds into test ( No RF)
    0.0000 MB /   1.00 sec =    0.0000 Mbps     0 retrans Tot:    76.8750 MB /  35.00 sec =   18.4250 Mbps     0 retrans 
    1.0000 MB /   1.00 sec =    8.3887 Mbps     0 retrans Tot:    77.8750 MB /  36.00 sec =   18.1462 Mbps     0 retrans
    4.9375 MB /   1.00 sec =   41.4178 Mbps     0 retrans Tot:    82.8125 MB /  37.00 sec =   18.7751 Mbps     0 retrans     
       ******************  Higher data rates here to maintain 20 Mbps average.
    4.8125 MB /   1.00 sec =   40.3709 Mbps     0 retrans Tot:    87.6250 MB /  38.00 sec =   19.3434 Mbps     0 retrans
    4.7500 MB /   1.00 sec =   39.8459 Mbps     0 retrans Tot:    92.3750 MB /  39.00 sec =   19.8691 Mbps     0 retrans
===========  skip forward
    2.3750 MB /   1.00 sec =   19.9202 Mbps     0 retrans Tot:   219.2500 MB /  92.00 sec =   19.9913 Mbps     0 retrans
    0.6250 MB /   1.00 sec =    5.2436 Mbps     0 retrans Tot:   219.8750 MB /  93.00 sec =   19.8327 Mbps     0 retrans
    0.0000 MB /   1.00 sec =    0.0000 Mbps     0 retrans Tot:   219.8750 MB /  94.00 sec =   19.6217 Mbps     0 retrans
    0.0000 MB /   1.00 sec =    0.0000 Mbps     0 retrans Tot:   219.8750 MB /  95.00 sec =   19.4152 Mbps     0 retrans
    0.0000 MB /   1.00 sec =    0.0000 Mbps     0 retrans Tot:   219.8750 MB /  96.00 sec =   19.2130 Mbps     0 retrans       
      *********************   60 seconds after first loss – another loss ( No RF)
    0.0000 MB /   1.00 sec =    0.0000 Mbps     0 retrans Tot:   219.8750 MB /  97.00 sec =   19.0149 Mbps     0 retrans
    0.0000 MB /   1.00 sec =    0.0000 Mbps     0 retrans Tot:   219.8750 MB /  98.00 sec =   18.8209 Mbps     0 retrans
    3.6250 MB /   1.00 sec =   30.4100 Mbps     0 retrans Tot:   223.5000 MB /  99.00 sec =   18.9379 Mbps     0 retrans
    4.8750 MB /   1.00 sec =   40.8921 Mbps     0 retrans Tot:   228.3750 MB / 100.00 sec =   19.1575 Mbps     0 retrans
=============  skip forward
    2.3750 MB /   1.00 sec =   19.9234 Mbps     0 retrans Tot:   359.9375 MB / 151.00 sec =   19.9959 Mbps     0 retrans
    2.3750 MB /   1.00 sec =   19.9228 Mbps     0 retrans Tot:   362.3125 MB / 152.00 sec =   19.9954 Mbps     0 retrans
    0.6250 MB /   1.00 sec =    5.2429 Mbps     0 retrans Tot:   362.9375 MB / 153.00 sec =   19.8990 Mbps     0 retrans
    0.0000 MB /   1.00 sec =    0.0000 Mbps     0 retrans Tot:   362.9375 MB / 154.00 sec =   19.7697 Mbps     0 retrans      
       ******************** 120 seconds later – another loss  ( No RF)
    0.0000 MB /   1.00 sec =    0.0000 Mbps     0 retrans Tot:   362.9375 MB / 155.00 sec =   19.6422 Mbps     0 retrans
    1.0625 MB /   1.00 sec =    8.9130 Mbps     0 retrans Tot:   364.0000 MB / 156.00 sec =   19.5734 Mbps     0 retrans
    4.8125 MB /   1.00 sec =   40.3708 Mbps     0 retrans Tot:   368.8125 MB / 157.00 sec =   19.7059 Mbps     0 retrans
=============  skip forward
    2.3750 MB /   1.00 sec =   19.9230 Mbps     0 retrans Tot:   505.3750 MB / 212.00 sec =   19.9971 Mbps     0 retrans
    0.6250 MB /   1.00 sec =    5.2429 Mbps     0 retrans Tot:   506.0000 MB / 213.00 sec =   19.9279 Mbps     0 retrans
    0.0000 MB /   1.00 sec =    0.0000 Mbps     0 retrans Tot:   506.0000 MB / 214.00 sec =   19.8347 Mbps     0 retrans
    0.0000 MB /   1.00 sec =    0.0000 Mbps     0 retrans Tot:   506.0000 MB / 215.00 sec =   19.7425 Mbps     0 retrans 
      ***********************    180 seconds later  - another loss  ( No RF)
    0.0000 MB /   1.00 sec =    0.0000 Mbps     0 retrans Tot:   506.0000 MB / 216.00 sec =   19.6511 Mbps     0 retrans
    4.7500 MB /   1.00 sec =   39.8438 Mbps     0 retrans Tot:   510.7500 MB / 217.00 sec =   19.7441 Mbps     0 retrans
    4.8750 MB /   1.00 sec =   40.8977 Mbps     0 retrans Tot:   515.6250 MB / 218.00 sec =   19.8412 Mbps     0 retrans
    
    SUMMARY
 572.2500 MB / 240.04 sec =   19.9986 Mbps 83 %TX 4 %RX 0 retrans 1.97 msRTT

Re: Packet loss/dropout with iperf3 - UDP

Posted: Fri Apr 05, 2019 6:51 pm
by mkirk
Is there a log file for the CYW43438 WiFi/BLE chip that shows fault events or other indications when it stops transmitting?

Re: Packet loss/dropout with iperf3 - UDP

Posted: Tue May 14, 2019 10:15 pm
by mkirk
Continue to see data rate drops. iperf3 test runs at 28 to 40 Mbps for hours then drops to 2-4 Mbps for 4 hours then recovers and goes back to 28-40 Mbps. What could cause this ? Test was running for 24 hours.