john_alan
Posts: 40
Joined: Mon Jul 22, 2019 7:20 pm

Pi keeps falling over and I don't know why

Wed Nov 20, 2019 8:49 am

So I've been trying to trouble-shoot this myself for weeks.

I have a Pi3 (and a Pi4, though the 4 causes no issues).



My Pi3 is fully up to date, on Raspbian Linux 4.19.81-v7+
This issue persisted prior to updates too.


The Pi serves the following tasks:

* Runs a cron job for a twitter bot
* Runs a push notification server for Apple PNS
* Runs PiHole (also acts as my DHCP and DNS server in my home)
* Executes a DynamicDNS job to update my domain registrar with my public IP
* SSH/VPN endpoint as I have a port forwarded to this device from my router

The load on the device is like 0.01 0.02 0.01 most of the time

Normally this all works perfectly for 3-6 days or so, then at some random point in time, I notice my MacBook or PC will stop working and it's clearly because the Pi is not serving DNS requests. I then notice that renewing a DNS lease does nothing, so the Pi has stopped serving these functions.

Also the Pi is not pingable and will not respond to SSH requests.

So I've been trawling through the logs and the unusual thing (to me) is that there are still items being printed during the "downtime", I have included the logs here below in a pastebin if anyone would like to view them.

https://pastebin.com/brV3k0nJ

Because my twitter bot runs every 10 mins, I know that this log above captures the 'falling over' about between 1am and 2am or so.

I did consider a hardware issue but I don't think it is one. It's really weird, can anyone help?
Last edited by john_alan on Wed Nov 20, 2019 8:53 am, edited 1 time in total.

ttsvetanov
Posts: 10
Joined: Tue Jun 12, 2018 9:04 am

Re: Pi keeps falling over and I don't know why

Wed Nov 20, 2019 8:51 am

Always start with memory card and PSU. Try to make image of your SD and copy it to a new memory card. Also try with another PSU.

john_alan
Posts: 40
Joined: Mon Jul 22, 2019 7:20 pm

Re: Pi keeps falling over and I don't know why

Wed Nov 20, 2019 8:54 am

OK I will look to replace them, is there anything you can see in the logs that's unusual?

john_alan
Posts: 40
Joined: Mon Jul 22, 2019 7:20 pm

Re: Pi keeps falling over and I don't know why

Wed Nov 20, 2019 9:01 am

To me this is the bit where things go bad:

Code: Select all

Nov 20 01:12:19 raspberrypi dhcpcd[466]: wlan0: carrier lost
Nov 20 01:12:19 raspberrypi dhcpcd[466]: wlan0: deleting address fe80::6bb:e56d:3d5a:8974
Nov 20 01:12:19 raspberrypi avahi-daemon[385]: Withdrawing address record for fe80::6bb:e56d:3d5a:8974 on wlan0.
Nov 20 01:12:19 raspberrypi avahi-daemon[385]: Leaving mDNS multicast group on interface wlan0.IPv6 with address fe80::6bb:e56d:3d5a:8974.
Nov 20 01:12:19 raspberrypi avahi-daemon[385]: Interface wlan0.IPv6 no longer relevant for mDNS.
Nov 20 01:12:19 raspberrypi avahi-daemon[385]: Withdrawing address record for 192.168.0.248 on wlan0.
Nov 20 01:12:19 raspberrypi avahi-daemon[385]: Leaving mDNS multicast group on interface wlan0.IPv4 with address 192.168.0.248.
Nov 20 01:12:19 raspberrypi avahi-daemon[385]: Interface wlan0.IPv4 no longer relevant for mDNS.
Nov 20 01:12:19 raspberrypi dhcpcd[466]: wlan0: deleting route to 192.168.0.0/24
Nov 20 01:12:19 raspberrypi dhcpcd[466]: wlan0: deleting default route via 192.168.0.1
Nov 20 01:12:19 raspberrypi wicd[383]: sending commands to master dhcpcd process
Nov 20 01:12:19 raspberrypi dhcpcd[31919]: sending commands to master dhcpcd process
Nov 20 01:12:19 raspberrypi dhcpcd[466]: control command: /sbin/dhcpcd -k wlan0
Nov 20 01:12:19 raspberrypi dhcpcd[466]: wlan0: removing interface
Nov 20 01:12:20 raspberrypi kernel: [344915.570800] IPv6: ADDRCONF(NETDEV_UP): wlan0: link is not ready
Nov 20 01:12:20 raspberrypi kernel: [344915.570988] brcmfmac: power management disabled
Nov 20 01:12:20 raspberrypi wicd[383]: sending commands to master dhcpcd process
Nov 20 01:12:20 raspberrypi dhcpcd[31941]: sending commands to master dhcpcd process
Nov 20 01:12:20 raspberrypi dhcpcd[466]: control command: /sbin/dhcpcd -k docker0
Nov 20 01:12:20 raspberrypi dhcpcd[466]: docker0: removing interface
Nov 20 01:12:20 raspberrypi avahi-daemon[385]: Withdrawing address record for 172.17.0.1 on docker0.
Nov 20 01:12:20 raspberrypi avahi-daemon[385]: Leaving mDNS multicast group on interface docker0.IPv4 with address 172.17.0.1.
Nov 20 01:12:20 raspberrypi avahi-daemon[385]: Interface docker0.IPv4 no longer relevant for mDNS.
Nov 20 01:12:20 raspberrypi kernel: [344915.833591] IPv6: ADDRCONF(NETDEV_UP): docker0: link is not ready
Nov 20 01:12:20 raspberrypi wicd[383]: Failed to connect to non-global ctrl_ifname: docker0  error: No such file or directory
Nov 20 01:12:24 raspberrypi kernel: [344919.963658] IPv6: ADDRCONF(NETDEV_UP): wlan0: link is not ready
Nov 20 01:12:24 raspberrypi kernel: [344919.963694] brcmfmac: power management disabled

DirkS
Posts: 10346
Joined: Tue Jun 19, 2012 9:46 pm
Location: Essex, UK

Re: Pi keeps falling over and I don't know why

Wed Nov 20, 2019 9:11 am

My Pi3 is fully up to date, on Raspbian Linux 4.19.81-v7+
That's not the current stable kernel version and I assume you ran 'rpi-update'.
That installs the development version of the kernel. If there is no specific reason to do that then I would suggest you go back to the stable kernel.

john_alan
Posts: 40
Joined: Mon Jul 22, 2019 7:20 pm

Re: Pi keeps falling over and I don't know why

Wed Nov 20, 2019 9:19 am

DirkS wrote:
Wed Nov 20, 2019 9:11 am
My Pi3 is fully up to date, on Raspbian Linux 4.19.81-v7+
That's not the current stable kernel version and I assume you ran 'rpi-update'.
That installs the development version of the kernel. If there is no specific reason to do that then I would suggest you go back to the stable kernel.
Thanks Dirk - yes I did update the Kernel, in an effort to resolve this issue, so this was occurring with the stable kernel too.

My Pi4 has similar logs, i.e. runs the same stack/firewalls etc, without issue.

The 3 just keeps losing it's network connection. I might try moving it to be wired.

User avatar
DarkPlatinum
Posts: 867
Joined: Thu Nov 02, 2017 2:30 pm
Location: Unknown
Contact: Website YouTube

Re: Pi keeps falling over and I don't know why

Wed Nov 20, 2019 9:23 am

john_alan wrote:
Wed Nov 20, 2019 9:19 am
The 3 just keeps losing it's network connection. I might try moving it to be wired.
Check your router, my router from time to time disables the 2.4Ghz band for no reason. Make sure the 2.4Ghz band has a different name to the 5Ghz band
1 * Raspberry Pi Zero W, 1 * Raspberry Pi 2, 1 * Raspberry Pi 3 1 * Raspberry Pi 3B + :mrgreen:

Check Out My Raspberry Site (Run on a Raspberry Pi 3B :) ): https://html.dynu.net

john_alan
Posts: 40
Joined: Mon Jul 22, 2019 7:20 pm

Re: Pi keeps falling over and I don't know why

Wed Nov 20, 2019 11:59 am

DarkPlatinum wrote:
Wed Nov 20, 2019 9:23 am
john_alan wrote:
Wed Nov 20, 2019 9:19 am
The 3 just keeps losing it's network connection. I might try moving it to be wired.
Check your router, my router from time to time disables the 2.4Ghz band for no reason. Make sure the 2.4Ghz band has a different name to the 5Ghz band

good idea, I just had a thought, could this issue be caused by aggressive 'dynamic' channel switching on my Virgin router?

User avatar
DarkPlatinum
Posts: 867
Joined: Thu Nov 02, 2017 2:30 pm
Location: Unknown
Contact: Website YouTube

Re: Pi keeps falling over and I don't know why

Wed Nov 20, 2019 5:36 pm

john_alan wrote:
Wed Nov 20, 2019 11:59 am
good idea, I just had a thought, could this issue be caused by aggressive 'dynamic' channel switching on my Virgin router?
That is somewhat possible but that would also mean that your other devices would be disconnecting as well. I know on the superhub 3 it comes with the same name for both frequencies. I had to change mine as I encountered issues.
1 * Raspberry Pi Zero W, 1 * Raspberry Pi 2, 1 * Raspberry Pi 3 1 * Raspberry Pi 3B + :mrgreen:

Check Out My Raspberry Site (Run on a Raspberry Pi 3B :) ): https://html.dynu.net

john_alan
Posts: 40
Joined: Mon Jul 22, 2019 7:20 pm

Re: Pi keeps falling over and I don't know why

Tue Dec 03, 2019 10:18 am

I already had a 2.4ghz and 5ghz network each, with unique SSIDs, however...


In the end this issue was caused by my Virgin Media "SuperHub 3.0" router with it's dynamic wifi channel switching.

I disabled it about 11 days ago and not a blip of an issue since.

This also explains why it occurred more often when I was using my laptop upstairs in my house, something which I thought was just my imagination, but obviously an action that forced the router to switch channels.

I don't know if there is a solution on the Pi side, it's just running the standard network stack, so I'm surprised this occurred.

User avatar
rpdom
Posts: 17022
Joined: Sun May 06, 2012 5:17 am
Location: Chelmsford, Essex, UK

Re: Pi keeps falling over and I don't know why

Tue Dec 03, 2019 10:29 am

Virgin "superhubs" are well know to have many problems. This is why I had to disable the router feature on my Superhub 3, put it into "modem" mode and connect it to another router to handle everything else.
Unreadable squiggle

Return to “Troubleshooting”