MuntyScruntfundle
Posts: 223
Joined: Fri Oct 27, 2017 11:14 pm

Finding differences in identical pis?

Sat Mar 02, 2019 1:42 pm

Hi there.

I have bunch of pis which are all built from the same image, they've have exactly the same installs, upgrades etc.

They all show different numbers of processes running and some take far longer to boot and shutdown than others.

Where do I start looking for issues? Are there boot and shutdown logs I can view easily?

Thanks.

epoch1970
Posts: 3557
Joined: Thu May 05, 2016 9:33 am
Location: Paris, France

Re: Finding differences in identical pis?

Sat Mar 02, 2019 2:05 pm

/var/log/syslog stil exists on Raspbian Stretch I believe?
Everything is in there, in theory.
"S'il n'y a pas de solution, c'est qu'il n'y a pas de problème." Les Shadoks, J. Rouxel

klricks
Posts: 6547
Joined: Sat Jan 12, 2013 3:01 am
Location: Grants Pass, OR, USA
Contact: Website

Re: Finding differences in identical pis?

Sat Mar 02, 2019 2:36 pm

Do all have the same SD card brand and size.

Are you sure all RPi models are the same? eg there are 2 versions of RPi 2 which are quite different.
Unless specified otherwise my response is based on the latest and fully updated Raspbian Buster w/ Desktop OS.

jahboater
Posts: 4604
Joined: Wed Feb 04, 2015 6:38 pm

Re: Finding differences in identical pis?

Sat Mar 02, 2019 2:46 pm

Network round trip times might vary?

Code: Select all

$ systemd-analyze blame
          7.651s dhcpcd.service
          1.647s dev-mmcblk0p2.device
          1.403s apt-daily-upgrade.service
          1.383s apt-daily.service
           716ms dphys-swapfile.service
           633ms keyboard-setup.service
           578ms systemd-fsck-root.service
           543ms raspi-config.service
           492ms systemd-logind.service
           444ms networking.service

gkaiseril
Posts: 636
Joined: Mon Aug 08, 2016 9:27 pm
Location: Chicago, IL

Re: Finding differences in identical pis?

Sat Mar 02, 2019 4:13 pm

You have verified all units are running the same OS using the terminal command 'uname -a'?

You can verify the cpu information on each Pi with the terminal command 'cat /proc/cpuinfo'?

The above command will also provide the type of CPU, CPU revision, and serial number for the Pi.

I expect you see some differences.


With the latest Raspbian you can use the command line command 'cat /proc/device-tree/model' to get the Pi model and revision as a string.
f u cn rd ths, u cn gt a gd jb n cmptr prgrmmng.

MuntyScruntfundle
Posts: 223
Joined: Fri Oct 27, 2017 11:14 pm

Re: Finding differences in identical pis?

Mon Mar 04, 2019 5:27 pm

They are identical. Same pi. Same cards. Same Distro. Same Updates. Same Upgrades. Same installs. Same mounts. Same Power. Same GPIO assignments.

The problem isn't with individual pis, especially the slow shutdown, it will randomly affect different pis on each system shutdown or reboot.

It's a software thing, I just don't know which bit of the system is stalling or failing. I can't ssh into these nodes when it happens, so it's my guess a service is trying to end, or do something, but I have not idea how to find out what it is.

There is indeed a /var/log/syslog, but it doesn't show very much of any use.

Thanks again folks.

MuntyScruntfundle
Posts: 223
Joined: Fri Oct 27, 2017 11:14 pm

Re: Finding differences in identical pis?

Mon Mar 04, 2019 5:27 pm

I suppose I could write a script that dumped the process list every second, but I'd have no control over when it would be closed down, so would it help??

MuntyScruntfundle
Posts: 223
Joined: Fri Oct 27, 2017 11:14 pm

Re: Finding differences in identical pis?

Mon Mar 04, 2019 5:31 pm

And should add: The network is a bunch of stacked GS752TS so there is no way (unless i decided to copy a 5gb file on stop) there can be any bottlenecks. And still this wouldn't explain why all the problem pis stay on for xx seconds longer, then all power off at the same time.

User avatar
scruss
Posts: 2419
Joined: Sat Jun 09, 2012 12:25 pm
Location: Toronto, ON
Contact: Website

Re: Finding differences in identical pis?

Mon Mar 04, 2019 5:35 pm

So these have network mounts? They can take a long (and sometimes variable) time to disconnect and allow the machine to shut down.
‘Remember the Golden Rule of Selling: “Do not resort to violence.”’ — McGlashan.

kaksi
Posts: 92
Joined: Tue Mar 10, 2015 6:19 am

Re: Finding differences in identical pis?

Wed Mar 06, 2019 9:48 am

I am surprised no one mentioned the MAC address.

ejolson
Posts: 3421
Joined: Tue Mar 18, 2014 11:47 am

Re: Finding differences in identical pis?

Wed Mar 06, 2019 5:22 pm

scruss wrote:
Mon Mar 04, 2019 5:35 pm
So these have network mounts? They can take a long (and sometimes variable) time to disconnect and allow the machine to shut down.
This seems possible to me. If all 128 Pi computers hit the same NFS server at the same time when shutting down, then the server better have been configured for that kind of load. What may happen is that the NFS server subsequently appears offline for some of the Pi's which then wait until the server recovers so they can unmount their network mounts.

I think there are a few parameters, including MAC address and a UUID related to the Bonjour networking protocol, that need to be different on each Pi. You may want distinct ssh keys as well. So if you cloned an already configured SD card by copying it to multiple SD cards, one for each Pi, there may be some parameters that are supposed to be randomly generated on first boot which are actually the same for each Pi. This sort of thing can also cause strange network behaviour.

Return to “General discussion”