norpi
Posts: 34
Joined: Wed Apr 08, 2015 4:50 pm

Grafana documents RPi keeps lolgging data while unreachable by network

Tue May 11, 2021 6:59 am

Hi

I run influxdb and grafana on my Rpi4 which is connected through wifi. I have set up a static ip.
Suddenly I can no longer access the grafana server or ping it. 1,5day passes and I rebooted it and everything is back to normal.
From the grafana plots I can see that all mentrics have been logging fine except the network speed which I log every 30 minutes. See attached screendump from Grafana showing that network tests stopped 05.09. at appx. 04PM/16:00.

A section of my syslog was prepared but I wasn't allowed to upload a .txt file as an attachment. the first lines of the log is pasted in below. Note that the 9th of may log starts in the middle of the file as it is probably currently being overwritten by todays events.

Any help would be much appreciated. I plan on having the rpi running for months to log temperatures in a large greenhouse and need the system to be operational at all times.

May 11 08:12:01 raspberrypi CRON[4237]: (root) CMD (/home/pi/rpi-speedtest-influx.py)
May 11 08:12:16 raspberrypi sSMTP[4235]: Unable to locate mail
May 11 08:12:16 raspberrypi cron[475]: sendmail: Cannot open mail:25
May 11 08:12:16 raspberrypi sSMTP[4235]: Cannot open mail:25
May 11 08:12:16 raspberrypi CRON[4225]: (root) MAIL (mailed 453 bytes of output but got status 0x0001 from MTA#012)
May 9 15:51:14 raspberrypi systemd-modules-load[117]: Inserted module 'i2c_dev'
May 9 15:51:14 raspberrypi fake-hwclock[114]: fr. 07. mai 18:43:46 +0000 2021
May 9 15:51:14 raspberrypi systemd-fsck[134]: rootfs: clean, 138501/467712 files, 1137930/1873920 blocks
May 9 15:51:14 raspberrypi systemd[1]: Started File System Check on Root Device.
May 9 15:51:14 raspberrypi systemd[1]: Starting Remount Root and Kernel File Systems...
May 9 15:51:14 raspberrypi systemd[1]: Started Set the console keyboard layout.
May 9 15:51:14 raspberrypi systemd[1]: Started Remount Root and Kernel File Systems.
May 9 15:51:14 raspberrypi systemd[1]: Starting Flush Journal to Persistent Storage...
May 9 15:51:14 raspberrypi systemd[1]: Starting Create System Users...
May 9 15:51:14 raspberrypi systemd[1]: Condition check resulted in Rebuild Hardware Database being skipped.
May 9 15:51:14 raspberrypi systemd[1]: Starting Load/Save Random Seed...
May 9 15:51:14 raspberrypi systemd[1]: Started udev Coldplug all Devices.
May 9 15:51:14 raspberrypi systemd[1]: Starting Helper to synchronize boot up for ifupdown...
May 9 15:51:14 raspberrypi systemd[1]: Started Flush Journal to Persistent Storage.
May 9 15:51:14 raspberrypi systemd[1]: Started Create System Users.
May 9 15:51:14 raspberrypi systemd[1]: Started Load/Save Random Seed.
May 9 15:51:14 raspberrypi systemd[1]: Started Helper to synchronize boot up for ifupdown.
May 9 15:51:14 raspberrypi systemd[1]: Starting Create Static Device Nodes in /dev...
May 9 15:51:14 raspberrypi systemd[1]: Started Create Static Device Nodes in /dev.
May 9 15:51:14 raspberrypi systemd[1]: Reached target Local File Systems (Pre).
May 9 15:51:14 raspberrypi systemd[1]: Starting udev Kernel Device Manager...
May 9 15:51:14 raspberrypi systemd[1]: Started udev Kernel Device Manager.
May 9 15:51:14 raspberrypi systemd[1]: Starting Show Plymouth Boot Screen...
Attachments
2 day data log.JPG
Two day data log Grafana
2 day data log.JPG (130.23 KiB) Viewed 353 times

User avatar
neilgl
Posts: 3250
Joined: Sun Jan 26, 2014 8:36 pm
Location: Near The National Museum of Computing

Re: Grafana documents RPi keeps lolgging data while unreachable by network

Tue May 11, 2021 9:40 am

Can you post the rpi-speedtest-influx.py
( My grafana/influxdb runs ok 24/7 )

norpi
Posts: 34
Joined: Wed Apr 08, 2015 4:50 pm

Re: Grafana documents RPi keeps lolgging data while unreachable by network

Tue May 11, 2021 11:29 am

The python script is posted under. I hve noticed a strange thing about the speed test; although i schedule this to run every 30minutes in cron, It seems there is new data in the grafana plot nearly every minute. I would expect the plot to be a flat indicating the same value for intervals of 30 minutes, instead it looks as if the script runs continously.

Please also have a look at the second screendump I just did, attached to this post where you can see how the CPU-load dropped after this mornings restart, the CPU-temp drops and the memory is freed up.

These are my only entries in crontab: (Edited by starting from terminal:"crontab -e")

*/30 * * * * /home/pi/rpi-speedtest-influx.py
*/5 * * * * /home/pi/rpi-stats-influx.py

rpi-speedtest-influx.py:
--------------------------------------
#!/usr/bin/env python

import datetime
import speedtest
from influxdb import InfluxDBClient

# influx configuration - edit these
ifuser = "grafanauser"
ifpass = "pass"
ifdb = "database"
ifhost = "myip"
ifport = 8086
measurement_name = "speedtest"

# take a timestamp for this measurement
time = datetime.datetime.utcnow()

# run a single-threaded speedtest using default server
s = speedtest.Speedtest()
s.get_best_server()
s.download(threads=1)
s.upload(threads=1)
res = s.results.dict()


# format the data as a single measurement for influx
body = [
{
"measurement": measurement_name,
"time": time,
"fields": {
"download": res["download"],
"upload": res["upload"],
"ping": res["ping"]
}
}
]

# connect to influx
ifclient = InfluxDBClient(ifhost,ifport,ifuser,ifpass,ifdb)

# write the measurement
ifclient.write_points(body)
Attachments
24hrs data log.JPG
24hrs data log.JPG (114.18 KiB) Viewed 276 times
Last edited by norpi on Tue May 11, 2021 9:12 pm, edited 1 time in total.

norpi
Posts: 34
Joined: Wed Apr 08, 2015 4:50 pm

Re: Grafana documents RPi keeps lolgging data while unreachable by network

Tue May 11, 2021 11:41 am

I think I have to study crontab a bit more. I just found out that I had edited both my users crontab and roots crontab and told cron to run the speestest in both where in roots crontab i had left the timing indication to * * * * * meaning run every minute. I guess this puts quite a lot of pressure on the system? Could it explain the "meltdown"?

User avatar
neilgl
Posts: 3250
Joined: Sun Jan 26, 2014 8:36 pm
Location: Near The National Museum of Computing

Re: Grafana documents RPi keeps lolgging data while unreachable by network

Mon May 17, 2021 7:25 pm

To edit root’s crontab we do sudo crontab -e
For user pi, when logged in as pi, just do crontab -e

Many have fallen into that trap…

Return to “Automation, sensing and robotics”