WANTED: Wifi testers!

[jamesh]

Hello all,

on the Pi linux kernel github there is a hacky fix, just merged, for a Wifi issue on PI3 and 0W we have been looking in to. If you are able to build your own kernel, have had Wifi problems in the past, and would like to test it, that would be extra cool.

Issue:
https://github.com/raspberrypi/linux/issues/1342

Kernel tree:
https://github.com/raspberrypi/linux
You need branch rpi-4.12.y

Instructions:
https://www.raspberrypi.org/documentati ... uilding.md

Those of you who are able to wait, we hope to have this back ported to 4.9 and accessible from rpi-update sometime next week, although this depends on various factors.

James

[BBUK]

Happy to help.

Unless you have any objections, I'll test on 4.9 and the Zero W as that matches a particular use case that I am looking to use internal wifi for. I can apply the change directly to the 4.9 source (it's only one line!)

BBUK

[Heater]

Ouch!

Before I leave my Pi 3 in a remote location today could you say, in short, what the issue is, what kernels/OS it relates to, what triggers it, and what is a work around?

I had a look in the issue but there is too much there to digest.

[BBUK]

Heater, I only had a quick scan myself but the TL;DR seems to be that some IP packets going through the brcmfmac driver have a priority setting. When that setting is at 0x10 or 0x08 some packets are silently lost, causing wifi issues. The PR (now merged with 4.12) forces the setting to 0x00. This is hence not a fix (which needs an analysis as to why the packets are being lost) but just a workaround.

PR is here so you can see what the hack is.

https://github.com/raspberrypi/linux/pull/2126

BBUK

[DougieLawson]

What does it fix? Is it all flavours of WiFi? Is it just the on-board WiFi on 3Bs/ZeroWs? Is it a specific dongle?

I've got three 3Bs that have on-board WiFi/BT
I've got a 2B that has the Redbear WiFi/BT pHAT
I've got a couple of Zeros with the "official" RPF USB dongle
I've got an A+ with RT5370
I've got a 1B with RT5370
I've got a spare RT5370 that's unused until RPi #15 arrives.

My 1B with Edimax RTL8188CUS isn't available for playing as I'm currently at 126 days into a full year of uptime.

It'll be interesting to see how fast a 3B with a WD hard drive can build a 4.12 kernel. But I'm not going to do that unless it's a useful test for you. I've not seen the underlying problem described at https://github.com/raspberrypi/linux/issues/1342 (or if I have seen it I've not noticed it).

[6by9]

https://github.com/raspberrypi/linux/is ... -316671120 is the details of the fix/workaround - there is an awful lot of fluff on that thread as it's been a dumping ground for any and every wifi issue.

Main gist is that we've found a mechanism whereby the Wifi firmware appears to be discarding a large number of packets if they have the priority flags set to maximise throughput or minimise latency. The workaround just kills off all priority at the wifi level.
The main question is whether this has any unintended consequences.

[jamesh]

I've changed the OP but to clarify, this is the onboard Wifi on the Pi3 and ZeroW. 6by9 has explained the actual issue, this hacky fix is a workaround for a possible bug in the wireless chip firmware (I cannot find anything wrong with the Linux driver). The firmware is closed so the bug is with the Cypress engineers (who now provide the chip) who are looking in to it.

[DougieLawson]

OK. I may go wild with a couple of 3Bs and my 2B with the Broacom based Redbear pHAT running 4.12 this weekend.

[dom]

To save rebuilding the kernel I've put instructions here to get a prebuilt module for testing.

[allfox]

I have a not quite useful test report.

I can't reproduce the problem at the very first place. Maybe I got a wrong setup?

I run hostapd on the Pi 3 on-board Wifi, and bridge it to a wired network.
A laptop Wifi connected to the Pi 3 AP.

Then ping from the Pi 3 to the laptop.

4.9.35-v7+

Code: Select all

allfox@pi1 ~ $ ping -c100 -Q0x10 lavender.local
PING lavender.local (10.157.193.47) 56(84) bytes of data.
64 bytes from lavender (10.157.193.47): icmp_seq=1 ttl=64 time=5.68 ms
64 bytes from lavender (10.157.193.47): icmp_seq=2 ttl=64 time=7.59 ms
64 bytes from lavender (10.157.193.47): icmp_seq=3 ttl=64 time=5.26 ms
64 bytes from lavender (10.157.193.47): icmp_seq=4 ttl=64 time=12.8 ms
64 bytes from lavender (10.157.193.47): icmp_seq=5 ttl=64 time=7.45 ms
64 bytes from lavender (10.157.193.47): icmp_seq=6 ttl=64 time=17.6 ms
64 bytes from lavender (10.157.193.47): icmp_seq=7 ttl=64 time=9.33 ms
64 bytes from lavender (10.157.193.47): icmp_seq=8 ttl=64 time=4.55 ms
64 bytes from lavender (10.157.193.47): icmp_seq=9 ttl=64 time=9.63 ms
64 bytes from lavender (10.157.193.47): icmp_seq=10 ttl=64 time=9.68 ms
64 bytes from lavender (10.157.193.47): icmp_seq=11 ttl=64 time=23.6 ms
64 bytes from lavender (10.157.193.47): icmp_seq=12 ttl=64 time=22.1 ms
64 bytes from lavender (10.157.193.47): icmp_seq=13 ttl=64 time=15.2 ms
64 bytes from lavender (10.157.193.47): icmp_seq=14 ttl=64 time=5.17 ms
64 bytes from lavender (10.157.193.47): icmp_seq=15 ttl=64 time=5.73 ms
64 bytes from lavender (10.157.193.47): icmp_seq=16 ttl=64 time=15.3 ms
64 bytes from lavender (10.157.193.47): icmp_seq=17 ttl=64 time=1.38 ms
64 bytes from lavender (10.157.193.47): icmp_seq=18 ttl=64 time=6.06 ms
64 bytes from lavender (10.157.193.47): icmp_seq=19 ttl=64 time=7.59 ms
64 bytes from lavender (10.157.193.47): icmp_seq=20 ttl=64 time=6.40 ms
64 bytes from lavender (10.157.193.47): icmp_seq=21 ttl=64 time=4.88 ms
64 bytes from lavender (10.157.193.47): icmp_seq=22 ttl=64 time=6.80 ms
64 bytes from lavender (10.157.193.47): icmp_seq=23 ttl=64 time=47.8 ms
64 bytes from lavender (10.157.193.47): icmp_seq=24 ttl=64 time=4.49 ms
64 bytes from lavender (10.157.193.47): icmp_seq=25 ttl=64 time=10.6 ms
64 bytes from lavender (10.157.193.47): icmp_seq=26 ttl=64 time=11.5 ms
64 bytes from lavender (10.157.193.47): icmp_seq=27 ttl=64 time=2.55 ms
64 bytes from lavender (10.157.193.47): icmp_seq=28 ttl=64 time=4.61 ms
64 bytes from lavender (10.157.193.47): icmp_seq=29 ttl=64 time=8.41 ms
64 bytes from lavender (10.157.193.47): icmp_seq=30 ttl=64 time=11.0 ms
64 bytes from lavender (10.157.193.47): icmp_seq=31 ttl=64 time=10.3 ms
64 bytes from lavender (10.157.193.47): icmp_seq=32 ttl=64 time=15.4 ms
64 bytes from lavender (10.157.193.47): icmp_seq=33 ttl=64 time=226 ms
64 bytes from lavender (10.157.193.47): icmp_seq=34 ttl=64 time=96.2 ms
64 bytes from lavender (10.157.193.47): icmp_seq=35 ttl=64 time=5.29 ms
64 bytes from lavender (10.157.193.47): icmp_seq=36 ttl=64 time=5.09 ms
64 bytes from lavender (10.157.193.47): icmp_seq=37 ttl=64 time=6.55 ms
64 bytes from lavender (10.157.193.47): icmp_seq=38 ttl=64 time=7.23 ms
64 bytes from lavender (10.157.193.47): icmp_seq=41 ttl=64 time=4.72 ms
64 bytes from lavender (10.157.193.47): icmp_seq=42 ttl=64 time=5.39 ms
64 bytes from lavender (10.157.193.47): icmp_seq=43 ttl=64 time=5.72 ms
64 bytes from lavender (10.157.193.47): icmp_seq=44 ttl=64 time=8.17 ms
64 bytes from lavender (10.157.193.47): icmp_seq=45 ttl=64 time=11.6 ms
64 bytes from lavender (10.157.193.47): icmp_seq=46 ttl=64 time=9.25 ms
64 bytes from lavender (10.157.193.47): icmp_seq=47 ttl=64 time=4.64 ms
64 bytes from lavender (10.157.193.47): icmp_seq=48 ttl=64 time=4.65 ms
64 bytes from lavender (10.157.193.47): icmp_seq=49 ttl=64 time=4.59 ms
64 bytes from lavender (10.157.193.47): icmp_seq=50 ttl=64 time=10.3 ms
64 bytes from lavender (10.157.193.47): icmp_seq=51 ttl=64 time=5.90 ms
64 bytes from lavender (10.157.193.47): icmp_seq=52 ttl=64 time=10.8 ms
64 bytes from lavender (10.157.193.47): icmp_seq=53 ttl=64 time=1.39 ms
64 bytes from lavender (10.157.193.47): icmp_seq=54 ttl=64 time=6.68 ms
64 bytes from lavender (10.157.193.47): icmp_seq=55 ttl=64 time=27.1 ms
64 bytes from lavender (10.157.193.47): icmp_seq=56 ttl=64 time=4.57 ms
64 bytes from lavender (10.157.193.47): icmp_seq=57 ttl=64 time=7.76 ms
64 bytes from lavender (10.157.193.47): icmp_seq=58 ttl=64 time=10.4 ms
64 bytes from lavender (10.157.193.47): icmp_seq=59 ttl=64 time=4.57 ms
64 bytes from lavender (10.157.193.47): icmp_seq=61 ttl=64 time=12.1 ms
64 bytes from lavender (10.157.193.47): icmp_seq=62 ttl=64 time=1.56 ms
64 bytes from lavender (10.157.193.47): icmp_seq=63 ttl=64 time=4.19 ms
64 bytes from lavender (10.157.193.47): icmp_seq=64 ttl=64 time=2.51 ms
64 bytes from lavender (10.157.193.47): icmp_seq=66 ttl=64 time=10.9 ms
64 bytes from lavender (10.157.193.47): icmp_seq=67 ttl=64 time=4.58 ms
64 bytes from lavender (10.157.193.47): icmp_seq=68 ttl=64 time=6.12 ms
64 bytes from lavender (10.157.193.47): icmp_seq=69 ttl=64 time=4.60 ms
64 bytes from lavender (10.157.193.47): icmp_seq=70 ttl=64 time=5.85 ms
64 bytes from lavender (10.157.193.47): icmp_seq=71 ttl=64 time=12.7 ms
64 bytes from lavender (10.157.193.47): icmp_seq=72 ttl=64 time=14.7 ms
64 bytes from lavender (10.157.193.47): icmp_seq=73 ttl=64 time=13.0 ms
64 bytes from lavender (10.157.193.47): icmp_seq=74 ttl=64 time=23.4 ms
64 bytes from lavender (10.157.193.47): icmp_seq=75 ttl=64 time=9.03 ms
64 bytes from lavender (10.157.193.47): icmp_seq=76 ttl=64 time=13.2 ms
64 bytes from lavender (10.157.193.47): icmp_seq=77 ttl=64 time=5.85 ms
64 bytes from lavender (10.157.193.47): icmp_seq=78 ttl=64 time=17.2 ms
64 bytes from lavender (10.157.193.47): icmp_seq=79 ttl=64 time=3.95 ms
64 bytes from lavender (10.157.193.47): icmp_seq=80 ttl=64 time=18.6 ms
64 bytes from lavender (10.157.193.47): icmp_seq=81 ttl=64 time=6.65 ms
64 bytes from lavender (10.157.193.47): icmp_seq=82 ttl=64 time=5.28 ms
64 bytes from lavender (10.157.193.47): icmp_seq=83 ttl=64 time=5.65 ms
64 bytes from lavender (10.157.193.47): icmp_seq=84 ttl=64 time=9.40 ms
64 bytes from lavender (10.157.193.47): icmp_seq=85 ttl=64 time=6.55 ms
64 bytes from lavender (10.157.193.47): icmp_seq=86 ttl=64 time=9.47 ms
64 bytes from lavender (10.157.193.47): icmp_seq=87 ttl=64 time=9.49 ms
64 bytes from lavender (10.157.193.47): icmp_seq=88 ttl=64 time=12.0 ms
64 bytes from lavender (10.157.193.47): icmp_seq=89 ttl=64 time=10.1 ms
64 bytes from lavender (10.157.193.47): icmp_seq=90 ttl=64 time=6.68 ms
64 bytes from lavender (10.157.193.47): icmp_seq=91 ttl=64 time=4.80 ms
64 bytes from lavender (10.157.193.47): icmp_seq=92 ttl=64 time=7.88 ms
64 bytes from lavender (10.157.193.47): icmp_seq=93 ttl=64 time=8.02 ms
64 bytes from lavender (10.157.193.47): icmp_seq=94 ttl=64 time=5.56 ms
64 bytes from lavender (10.157.193.47): icmp_seq=95 ttl=64 time=29.0 ms
64 bytes from lavender (10.157.193.47): icmp_seq=96 ttl=64 time=113 ms
64 bytes from lavender (10.157.193.47): icmp_seq=97 ttl=64 time=5.47 ms
64 bytes from lavender (10.157.193.47): icmp_seq=98 ttl=64 time=8.60 ms
64 bytes from lavender (10.157.193.47): icmp_seq=99 ttl=64 time=140 ms

--- lavender.local ping statistics ---
100 packets transmitted, 95 received, 5% packet loss, time 99115ms
rtt min/avg/max/mdev = 1.386/14.960/226.236/29.668 ms
allfox@pi1 ~ $

4.12.2-v7

Code: Select all

allfox@pi1 ~ $ ping -c100 -Q 0x10 lavender.local
PING lavender.local (10.157.193.47) 56(84) bytes of data.
64 bytes from lavender (10.157.193.47): icmp_seq=1 ttl=64 time=4.21 ms
64 bytes from lavender (10.157.193.47): icmp_seq=2 ttl=64 time=4.90 ms
64 bytes from lavender (10.157.193.47): icmp_seq=3 ttl=64 time=4.63 ms
64 bytes from lavender (10.157.193.47): icmp_seq=4 ttl=64 time=12.2 ms
64 bytes from lavender (10.157.193.47): icmp_seq=5 ttl=64 time=8.69 ms
64 bytes from lavender (10.157.193.47): icmp_seq=6 ttl=64 time=9.84 ms
64 bytes from lavender (10.157.193.47): icmp_seq=7 ttl=64 time=5.21 ms
64 bytes from lavender (10.157.193.47): icmp_seq=8 ttl=64 time=15.1 ms
64 bytes from lavender (10.157.193.47): icmp_seq=9 ttl=64 time=14.8 ms
64 bytes from lavender (10.157.193.47): icmp_seq=10 ttl=64 time=182 ms
64 bytes from lavender (10.157.193.47): icmp_seq=12 ttl=64 time=8.26 ms
64 bytes from lavender (10.157.193.47): icmp_seq=13 ttl=64 time=9.83 ms
64 bytes from lavender (10.157.193.47): icmp_seq=16 ttl=64 time=343 ms
64 bytes from lavender (10.157.193.47): icmp_seq=17 ttl=64 time=4.86 ms
64 bytes from lavender (10.157.193.47): icmp_seq=18 ttl=64 time=32.0 ms
64 bytes from lavender (10.157.193.47): icmp_seq=19 ttl=64 time=10.5 ms
64 bytes from lavender (10.157.193.47): icmp_seq=20 ttl=64 time=15.4 ms
64 bytes from lavender (10.157.193.47): icmp_seq=21 ttl=64 time=5.08 ms
64 bytes from lavender (10.157.193.47): icmp_seq=22 ttl=64 time=4.77 ms
64 bytes from lavender (10.157.193.47): icmp_seq=23 ttl=64 time=10.5 ms
64 bytes from lavender (10.157.193.47): icmp_seq=24 ttl=64 time=5.34 ms
64 bytes from lavender (10.157.193.47): icmp_seq=25 ttl=64 time=4.74 ms
64 bytes from lavender (10.157.193.47): icmp_seq=26 ttl=64 time=5.16 ms
64 bytes from lavender (10.157.193.47): icmp_seq=27 ttl=64 time=11.3 ms
64 bytes from lavender (10.157.193.47): icmp_seq=28 ttl=64 time=4.74 ms
64 bytes from lavender (10.157.193.47): icmp_seq=29 ttl=64 time=7.82 ms
64 bytes from lavender (10.157.193.47): icmp_seq=30 ttl=64 time=7.92 ms
64 bytes from lavender (10.157.193.47): icmp_seq=32 ttl=64 time=5.93 ms
64 bytes from lavender (10.157.193.47): icmp_seq=33 ttl=64 time=15.5 ms
64 bytes from lavender (10.157.193.47): icmp_seq=34 ttl=64 time=5.35 ms
64 bytes from lavender (10.157.193.47): icmp_seq=35 ttl=64 time=7.18 ms
64 bytes from lavender (10.157.193.47): icmp_seq=36 ttl=64 time=21.9 ms
64 bytes from lavender (10.157.193.47): icmp_seq=37 ttl=64 time=9.85 ms
64 bytes from lavender (10.157.193.47): icmp_seq=38 ttl=64 time=21.7 ms
64 bytes from lavender (10.157.193.47): icmp_seq=39 ttl=64 time=4.81 ms
64 bytes from lavender (10.157.193.47): icmp_seq=41 ttl=64 time=12.7 ms
64 bytes from lavender (10.157.193.47): icmp_seq=42 ttl=64 time=28.1 ms
64 bytes from lavender (10.157.193.47): icmp_seq=43 ttl=64 time=6.58 ms
64 bytes from lavender (10.157.193.47): icmp_seq=44 ttl=64 time=6.74 ms
64 bytes from lavender (10.157.193.47): icmp_seq=45 ttl=64 time=4.83 ms
64 bytes from lavender (10.157.193.47): icmp_seq=46 ttl=64 time=37.4 ms
64 bytes from lavender (10.157.193.47): icmp_seq=47 ttl=64 time=5.83 ms
64 bytes from lavender (10.157.193.47): icmp_seq=48 ttl=64 time=27.8 ms
64 bytes from lavender (10.157.193.47): icmp_seq=49 ttl=64 time=4.74 ms
64 bytes from lavender (10.157.193.47): icmp_seq=50 ttl=64 time=9.94 ms
64 bytes from lavender (10.157.193.47): icmp_seq=52 ttl=64 time=4.85 ms
64 bytes from lavender (10.157.193.47): icmp_seq=53 ttl=64 time=165 ms
64 bytes from lavender (10.157.193.47): icmp_seq=55 ttl=64 time=4.88 ms
64 bytes from lavender (10.157.193.47): icmp_seq=56 ttl=64 time=10.8 ms
64 bytes from lavender (10.157.193.47): icmp_seq=57 ttl=64 time=11.3 ms
64 bytes from lavender (10.157.193.47): icmp_seq=59 ttl=64 time=281 ms
64 bytes from lavender (10.157.193.47): icmp_seq=60 ttl=64 time=6.28 ms
64 bytes from lavender (10.157.193.47): icmp_seq=62 ttl=64 time=7.59 ms
64 bytes from lavender (10.157.193.47): icmp_seq=63 ttl=64 time=9.41 ms
64 bytes from lavender (10.157.193.47): icmp_seq=64 ttl=64 time=8.21 ms
64 bytes from lavender (10.157.193.47): icmp_seq=65 ttl=64 time=12.7 ms
64 bytes from lavender (10.157.193.47): icmp_seq=66 ttl=64 time=4.75 ms
64 bytes from lavender (10.157.193.47): icmp_seq=67 ttl=64 time=19.9 ms
64 bytes from lavender (10.157.193.47): icmp_seq=68 ttl=64 time=31.2 ms
64 bytes from lavender (10.157.193.47): icmp_seq=69 ttl=64 time=5.09 ms
64 bytes from lavender (10.157.193.47): icmp_seq=70 ttl=64 time=10.2 ms
64 bytes from lavender (10.157.193.47): icmp_seq=71 ttl=64 time=5.43 ms
64 bytes from lavender (10.157.193.47): icmp_seq=72 ttl=64 time=7.87 ms
64 bytes from lavender (10.157.193.47): icmp_seq=73 ttl=64 time=12.0 ms
64 bytes from lavender (10.157.193.47): icmp_seq=74 ttl=64 time=20.1 ms
64 bytes from lavender (10.157.193.47): icmp_seq=75 ttl=64 time=64.7 ms
64 bytes from lavender (10.157.193.47): icmp_seq=76 ttl=64 time=11.9 ms
64 bytes from lavender (10.157.193.47): icmp_seq=77 ttl=64 time=4.65 ms
64 bytes from lavender (10.157.193.47): icmp_seq=78 ttl=64 time=4.78 ms
64 bytes from lavender (10.157.193.47): icmp_seq=79 ttl=64 time=6.41 ms
64 bytes from lavender (10.157.193.47): icmp_seq=80 ttl=64 time=20.7 ms
64 bytes from lavender (10.157.193.47): icmp_seq=81 ttl=64 time=11.6 ms
64 bytes from lavender (10.157.193.47): icmp_seq=82 ttl=64 time=9.25 ms
64 bytes from lavender (10.157.193.47): icmp_seq=83 ttl=64 time=6.68 ms
64 bytes from lavender (10.157.193.47): icmp_seq=84 ttl=64 time=21.7 ms
64 bytes from lavender (10.157.193.47): icmp_seq=85 ttl=64 time=4.66 ms
64 bytes from lavender (10.157.193.47): icmp_seq=86 ttl=64 time=7.53 ms
64 bytes from lavender (10.157.193.47): icmp_seq=87 ttl=64 time=12.8 ms
64 bytes from lavender (10.157.193.47): icmp_seq=88 ttl=64 time=7.40 ms
64 bytes from lavender (10.157.193.47): icmp_seq=89 ttl=64 time=5.46 ms
64 bytes from lavender (10.157.193.47): icmp_seq=90 ttl=64 time=4.92 ms
64 bytes from lavender (10.157.193.47): icmp_seq=91 ttl=64 time=4.62 ms
64 bytes from lavender (10.157.193.47): icmp_seq=92 ttl=64 time=24.9 ms
64 bytes from lavender (10.157.193.47): icmp_seq=93 ttl=64 time=4.89 ms
64 bytes from lavender (10.157.193.47): icmp_seq=94 ttl=64 time=13.0 ms
64 bytes from lavender (10.157.193.47): icmp_seq=95 ttl=64 time=13.0 ms
64 bytes from lavender (10.157.193.47): icmp_seq=96 ttl=64 time=9.81 ms
64 bytes from lavender (10.157.193.47): icmp_seq=97 ttl=64 time=15.1 ms
64 bytes from lavender (10.157.193.47): icmp_seq=98 ttl=64 time=13.2 ms
64 bytes from lavender (10.157.193.47): icmp_seq=99 ttl=64 time=16.2 ms

--- lavender.local ping statistics ---
100 packets transmitted, 90 received, 10% packet loss, time 98995ms
rtt min/avg/max/mdev = 4.217/21.696/343.184/51.001 ms
allfox@pi1 ~ $

[DougieLawson]

To save rebuilding the kernel I've put instructions here to get a prebuilt module for testing.

Hey, there goes my weekend project. May go to 4.12 anyway on one of my Raspbian Stretch machines.

[6by9]

@allfox: I suspect running as an ap changes the behaviour. Thanks for testing though - recreating the issue has been half the battle.

[dom]

Latest rpi-update kernel now includes the BCDC priority patch.