davef
Posts: 64
Joined: Wed Feb 20, 2013 8:39 pm
Location: Christchurch, NZ

Switching wlan0 connections takes a long time

Fri Dec 28, 2018 7:50 pm

On a RaspPi 3 with an up to date Jessie distro.

I have one network (wlan0) which the Pi uses dhcp to connect to a hotspot
and on the same network (wlan0) where I use a static IP to connect to a
ESP8266. As per tutorials on modifying the dhcpcd.conf file.

When I switch from the dhcp managed connection to the static connection
it can take up 30 seconds for it to acquire the network. I determined
this by repeatedly pinging it. Switching from the static network connection to
the dhcp managed one happens immediately.

To switch between the two devices I swap /etc/interface files, which call
specific wpa_supplicant configs and then restart the network.

Any pointers to how I can speed this up?

epoch1970
Posts: 5732
Joined: Thu May 05, 2016 9:33 am
Location: Paris, France

Re: Switching wlan0 connections takes a long time

Wed Jan 02, 2019 3:43 pm

The DHCP part can only succeed after authentication has completed.
Either can experience slow-downs, since these are negotiation processes. Poor WiFi reception will compound the issue.

A sure way to get a slow response from the DHCP server (or no IP at all) is to start the DHCP client before the interface has authenticated with its AP.
A DHCP client starts by requesting for a lease multiple times in quick succession, then it backs-off and keeps requesting at a much slower pace, like 1 request every 20 secs. (Sometimes it also gives up altogether after some delay.)
Not starting the dhcp client first, or adding a delay before the first request might accelerate the overall process...

To pinpoint the exact reason try checking config files/processes to verify the activation order, and also look at the system log.
"S'il n'y a pas de solution, c'est qu'il n'y a pas de problème." Les Shadoks, J. Rouxel

davef
Posts: 64
Joined: Wed Feb 20, 2013 8:39 pm
Location: Christchurch, NZ

Re: Switching wlan0 connections takes a long time

Wed Jan 02, 2019 6:41 pm

epoch1970,

Thank you for your response. Here is a snippet of syslog
Jan 2 23:59:14 raspberrypi networking[31505]: DHCPDISCOVER on wlan0 to 255.255.255.255 port 67 interval 17
Jan 2 23:59:16 raspberrypi wpa_supplicant[31558]: wlan0: Trying to associate with 1c:af:05:39:1a:93 (SSID='AndroidAP8362' freq=2437 MHz)
Jan 2 23:59:16 raspberrypi wpa_supplicant[31558]: wlan0: Association request to the driver failed
Jan 2 23:59:16 raspberrypi kernel: [383770.475403] IPv6: ADDRCONF(NETDEV_CHANGE): wlan0: link becomes ready
Jan 2 23:59:16 raspberrypi wpa_supplicant[31558]: wlan0: Associated with 1c:af:05:39:1a:93
Jan 2 23:59:16 raspberrypi wpa_supplicant[31558]: wlan0: WPA: Key negotiation completed with 1c:af:05:39:1a:93 [PTK=CCMP GTK=CCMP]
Jan 2 23:59:16 raspberrypi wpa_supplicant[31558]: wlan0: CTRL-EVENT-CONNECTED - Connection to 1c:af:05:39:1a:93 completed [id=0 id_str=]
As the on-board WiFi died I am using a ASUS N10 WiFi dongle. I had issues using that dongle in another system so maybe the comment above
Association request to the driver failed
is slowing things down further through the process.

I have another new Pi3, so will try that.

Cheers,
Dave

epoch1970
Posts: 5732
Joined: Thu May 05, 2016 9:33 am
Location: Paris, France

Re: Switching wlan0 connections takes a long time

Thu Jan 03, 2019 3:39 pm

So the dhcp client starts before the interface is authenticated...
I would try to see how to add a delay or how to have the dhcp client start only once authenticated.

This is software and setup dependent, so giving a precise answer is not easy. However since you’re dealing with Jessie and ifupdown, I could suggest the following (from memory, I am away from my Pis):
1) set the interface as “manual” and not dhcp in interfaces. That will block ifupdown from launching the dhcp client.
2a) modify the wpa_supplicant script that dwells in /etc/network/if-pre.d/, or
2b) add a “dhcp-client” script in if-pre.d and create a new custom stanza in interfaces, like “pre-up dhcp-now” that the script will parse and react to
3) From either script, check the WPA status of the interface and delay calling the dhcp client until the status reads “connected”. The wpa_cli executable can be used to check status, alternatively wpa_supplicant can support an “action script” (option -a) that gets called all along the lifecycle of the interface (up, connected, disconnected...)

If you look at how ifupdown scripts work, it is not too difficult to come up with your own processing logic. Caveat: ifupdown is unaware of netlink events, it will lock-up happily waiting forever in a script... Your scripts have to control the state machine (not simple), so the simpler the mod, the better.

HTH
"S'il n'y a pas de solution, c'est qu'il n'y a pas de problème." Les Shadoks, J. Rouxel

davef
Posts: 64
Joined: Wed Feb 20, 2013 8:39 pm
Location: Christchurch, NZ

Re: Switching wlan0 connections takes a long time

Thu Jan 03, 2019 11:08 pm

Hi epoch1970,

Thanks for the added information.

After your comments, I think the snippet I have included is not pointing to the real problem. I saw
Association request to the driver failed
as the likely issue, possibly an incorrect assumption.

The Android AP8362 is on the dhcp managed network and comes up quickly, the ESP8266 is on manual network seems to be the one that takes about 30 seconds to come up. Even though I get the driver fail error the script suggests that no time is wasted bringing the dhcp managed connection up.

dhcpcd is running to manage the static connection, could there be some conflict between this and dhcp?

You have given me a few things to investigate.

Regards,
Dave

epoch1970
Posts: 5732
Joined: Thu May 05, 2016 9:33 am
Location: Paris, France

Re: Switching wlan0 connections takes a long time

Fri Jan 04, 2019 12:34 am

Oops. I totally misread your initial message.
My apologies.

Reboot: DHCP + AP8362 authentication goes fast, static IP + ESP8266 authentication goes slow.

I can’t see any reason why static IP assignment could be slow. There is no negociation there, it is a local process, it should be instantaneous. (Since you do ping in the end I don’t think there could be an error in the static IP config, but maybe you want to double-check it)

This leaves us with the WPA authentication part.
I’m not clear on the sequence of events in a WPA handshake and on the key renewal process, but could it be possible that wpa_supplicant tries to reuse the keys it shared with the previous AP when connecting to the new one? (Seems unlikely if this is a new wpa_supplicant process)

Or maybe that alternate AP is just very slow to react.
Are connections to the ESP access point always slow? After the first connection, if wlan0 goes down and up again, does it take 30 secs for connectivity to be effective again?

(This is very strange to me.. had I read your OP correctly, I don’t think I would have ventured an answer)
"S'il n'y a pas de solution, c'est qu'il n'y a pas de problème." Les Shadoks, J. Rouxel

davef
Posts: 64
Joined: Wed Feb 20, 2013 8:39 pm
Location: Christchurch, NZ

Re: Switching wlan0 connections takes a long time

Fri Jan 04, 2019 3:02 am

No apologies necessary. As I do not understand the dhcp/dhcpcd network acquisition process I will likely post incorrect clues to help solving it!

Maybe some more information on the ESP8266. My understanding it is operating in "beacon mode" and is operating "open" key_management.. It is part of a solar battery management system. It only sends out data once every few seconds and is on a fixed IP.

Maybe, dhcpcd on the Pi can not negotiate to achieve a proper connection and times-out after 30 seconds. This is only a guess on my part.

Dave

epoch1970
Posts: 5732
Joined: Thu May 05, 2016 9:33 am
Location: Paris, France

Re: Switching wlan0 connections takes a long time

Fri Jan 04, 2019 11:42 am

Ok I think I got it, finally.
Please confirm every single statement below:
- This Pi is running Jessie, not the newer Stretch
- You want to switch the Pi from AP 1 to AP 2
- Since the WiFi client config file to use by the Pi is specified in /etc/network/interfaces, you flip-flop this file to target one AP or the other
- AP 1 offers only DHCP, AP 2 does not at all
- You’re using dhcpcd’s fallback mechanism to detect the presence of AP 1, and fall back to a static IP config fit for AP 2 if AP 1 fails to deliver a dhcp lease. Fallback takes 30 secs, too much for your application.

If this is all correct, and while I am fairly confused about how “fallback” really works, in “man dhcpcd.conf”, I have seen the following:
timeout seconds
Timeout after seconds, instead of the default 30. A setting of 0 seconds causes dhcpcd to wait forever to get a lease. If dhcpcd is working on a single interface then dhcpcd will exit when a timeout occurs, otherwise dhcpcd will fork into the background. If using IPv4LL then dhcpcd start the IPv4LL process after the timeout and then wait a little longer before really timing out.
What about setting “timeout 1” in dhcpcd.conf, and checking if the ping delay over AP2’s network disappears?
(With timeout 1, unless AP 1 is lightning fast and WiFi coverage perfect, getting a lease from AP1 will always fail. This is just for testing)
"S'il n'y a pas de solution, c'est qu'il n'y a pas de problème." Les Shadoks, J. Rouxel

davef
Posts: 64
Joined: Wed Feb 20, 2013 8:39 pm
Location: Christchurch, NZ

Re: Switching wlan0 connections takes a long time

Fri Jan 04, 2019 10:30 pm

Hi epoch 1970,

1) Jessie latest upgrades applied
2) Yes
3) Yes
4) Yes, that is my understanding
5) I guess so, I am not confident how this part works.

I won't be back at the remote site for a few days to try your suggestion, regarding timeout in dhcpcd.conf

I will also read this page carefully:
https://wiki.archlinux.org/index.php/dhcpcd
especially fallback profile and the Tips section.

You have been extremely helpful, I really appreciate it.

Regards,
Dave

epoch1970
Posts: 5732
Joined: Thu May 05, 2016 9:33 am
Location: Paris, France

Re: Switching wlan0 connections takes a long time

Sat Jan 05, 2019 12:00 pm

Ok. So here is what I have in mind, in case the fallback thing proves to be the culprit and it needs to go.
This would AFAIK only work with Jessie. Not applicable to Stretch.

- Remove package dhcdpc5 from the system. All networking config needs then to be done old-school, in /etc/network/interfaces*
- Add eth0 to file interfaces as needed, remove wlan0 specification altogether.
- Create full configs in /etc/network/interfaces.d/ for wlan0 on AP1 and for AP2. I don’t recall the naming convention that you’ll need to break so that by default ifupdown will not take these (conflicting) files into account. Perhaps adding a suffix does the trick, e.g. “wlan0-ap1.in”
For AP1 the file would read “iface wlan0 inet dhcp” on the second line, for AP2 that line would be “iface wlan0 inet manual” (followed by the static IP specification)
- Adapt if needed the current file flip-flop mechanism to “ifdown wlan0”, delete/create the symbolic link /etc/interfaces.d/wlan0 pointing to whichever interface definition you want at that moment, and “ifup wlan0”

If think that should do the trick nicely and repeatably.

(Dhcpcd is also scriptable and supports “env” variables. You could look at the dhcpcd-hooks mechanism and default scripts (in /var/lib/dhcpcd-hooks.d/ ?), if you prefer to do away with ifupdown.
I think this is the more complex route, not sure it would work on Jessie. Dhcpcd reacts to netlink events, so if a hook script does something like “ip link xxx down”, you might get yourself caught in a loop...)

Good luck, have fun.
"S'il n'y a pas de solution, c'est qu'il n'y a pas de problème." Les Shadoks, J. Rouxel

bzt
Posts: 590
Joined: Sat Oct 14, 2017 9:57 pm

Re: Switching wlan0 connections takes a long time

Sat Jan 05, 2019 6:53 pm

TBH, it's very strange that the static connection needs more time.

I think you should first determine where the latency coming from: on your machine with the interface configuration, or between your interface and the gateway. In another terminal try to run "watch 1 ip addr sh". This will report your interfaces (whether they are up/down, which IP they got etc.), refreshing the list every sec (you can exit by pressing Ctrl+C). If your interface with static IP is up and running quickly, but you can't ping your gateway, then I suspect this might be an ISO/OSI L2 issue. You should delete the ARP table on the client (and probably on your router too, as that has cached an old, dhcp-served IP with the RPi's MAC address). Again, I'm not sure if this is the cause, just a guess, but clearing the ARP tables worth a try.

Second, you can tweek network timeouts with sysctl. List them with "sysctl -a". Change with "sysctl -w key=value". But be careful, it's easy to mess up networking by sysctl. I'd suggest to look for a ttl value of 30, and always read about the key before you change it. You should try to set "arp_announce" and "arp_notify" too to broadcast your new IP to the gateway. When your result is ok, only then make the change permanent by "sysctl -p". I would say this advice is for experts only (but may very well solve your latency issue), so be careful.

Good luck,
bzt

davef
Posts: 64
Joined: Wed Feb 20, 2013 8:39 pm
Location: Christchurch, NZ

Re: Switching wlan0 connections takes a long time

Sun Jan 06, 2019 7:23 pm

Thanks both for further suggestions. I will digest the comments, there is quite a bit of new content in them that I have to get my head around.

Progress after the dhcpcd.conf timeout to 1 second suggestion by epoch 1970:

Changed timeout in dhcpcd.conf to 1 second. All I can tell at this
stage is that the system still works. As far as I am aware, one can not
check that the network is up and running by checking any status codes
from /etc/init.d networking restart.

Here is the Bash script snippet that I use to check that the network
is up and running.

Code: Select all

bring up the SBMS (static) interface
cp /etc/network/interfaces_sbms /etc/network/interfaces
service networking restart

while [ $number_of_ping_attempts -lt 5 ]
do
  ping -c1 -w5 192.168.4.1 &> /dev/null

  if [ $?  == 0 ]
  then
    echo "connected to the SBMS" >> /home/mail.log

    break
  else
    echo "NOT connected to the SBMS" >> /home/mail.log

    sleep 10

    number_of_ping_attempts=$[$number_of_ping_attempts+1]
  fi
done
After a few days or weeks I will have some idea of whether a change is
successful by the number of "NOT connected to the SBMS" entries
in my mail.log file.

Dave

davef
Posts: 64
Joined: Wed Feb 20, 2013 8:39 pm
Location: Christchurch, NZ

Re: Switching wlan0 connections takes a long time

Mon Jan 07, 2019 10:16 pm

bzt,

Had a look at "sysctl -a" and thought I don't want to mess around
at this level. The suggestion of using "watch 1 ip addr sh" ... at the
moment I don't know how to run the 2nd terminal on my hardware. I hope
it would give similar clues as looking at syslog.

epoch1970,

Using your suggestions for running "old-school" static "iface wlan0 inet static"
I get stuck where "ifconfig" says I have assigned a static address to AP2 and
"route" says I have a legal route to it, but can not ping it (network
unreachable). This was my previous experience and the only way
I could get it to work was "iface wlan0 inet dhcp" instead of static.

I assume that AP1 must be giving an IP to my RaspberryPi for the AP2
network. As far as I am aware the Pi is not running dhcp, that is
what "service --status-all" tells me.

I also used this tutorial:
https://www.modmypi.com/blog/tutorial-h ... ip-address
but noted that for Jessie you needed this update:
https://www.modmypi.com/blog/how-to-giv ... ess-update
that brings us back to dhcpcd.conf

When using dhcpcd I think you have to be careful what ends up in
/var/lib/dhcp/ when you are "playing around". I had multiple old (?)
entries in it.

I have two working methods now and learned a few things in the process.

Thanks guys,
Dave

Return to “Networking and servers”