notro
Posts: 695
Joined: Tue Oct 16, 2012 6:21 pm
Location: Drammen, Norway

Under-voltage event logging?

Tue Feb 06, 2018 9:34 pm

Inadequate power supplies seem to be a prevailing challenge for both new (and old) Raspberry Pi users.
As the Pi's become more powerful they also require more power, so the power supply that worked with the Pi2, could result in under-voltage warnings on the Pi3.

Those with a display connected get an indication that something is up with the lightning symbol, and those that run headless see the power led turning off.
Unless one pays attention to those indicators, there is no way to tell when under-voltage happened except that 'vcgencmd get_throttled' can tell that it did happen at some point during this run.

I have wanted for some time to improve on this situation, by finding a way to log these events.
So I figured I should just allocate some time to this and see if it was possible to find a solution that could be included in Raspbian.

Since I mainly do kernel development, my first impulse was to do it in the kernel, especially since I wanted the event to be communicated across a reboot.
If an under-voltage event results in a kernel crash and the watchdog ends up resetting, it would be nice to know about that.
It turned out that the status register which is preserved across reboots is fully utilised, so one of the reasons for doing it in the kernel fell away.
Another thing is that there's currently no (easy) way to do this except by polling, so it might as well be done in userspace.
Ref: https://github.com/raspberrypi/linux/issues/2367

My current idea is to write a systemd service in python that polls get_throttled and reports the event to the kernel log. It will also write to a file which will be read on startup so we can write a message to the log that an under-voltage event happened during the previous run.

There are couple thing that needs to be decided regardless of logging implementation:

1. How often should we poll

This depends on the cost of doing polling vs. the precision we want. Does anyone know if the firmware itself polls to find out? If so it would be nice to know it's polling period.

2. What kind of events should we have

Since we have to poll, we loose precision. We can know two things: That an event has happened and whether or not it is happening right now.
get_throttled bits:
0: under-voltage
When set, we report under-voltage and then report when it's over. Every time? I guess we need some threshold to avoid spamming the log.
16: under-voltage has occurred
Bit set but we haven't seen bit 0 set, report under-voltage event. Probably distinguish between one detected in the first poll on startup and one that happens between two polls.

3. Where to log

I assumed the kernel log for an event like this, but I don't know where people generally go looking if something strange/erroneous is happening.

Lightning bolt and vc4:
The firmware is able to put an icon on the display because it knows where the framebuffer is in memory (and it's layout) since it handles allocation on behalf of the fbdev driver. When using the vc4 driver, the firmware doesn't know where the buffer is, so it can't put up a symbol.
In time we will move to vc4, so at some point we need a new way to communicate under-voltage to the desktop.

It would be nice to get some feedback on this project to get a broader picture before trying an implementation.
Maybe some simple lightweight monitoring framework already exists that can be utilised for this purpose.

MaxK1
Posts: 1043
Joined: Sun Aug 26, 2012 11:34 pm

Re: Under-voltage event logging?

Wed Feb 07, 2018 4:26 pm

I think vcgencmd get_throttled only shows that it happened at least once.
You are in a maze of twisty little passages, all alike.
When General Failure and Major Disaster get together, Private Parts usually suffers.

PiGraham
Posts: 3706
Joined: Fri Jun 07, 2013 12:37 pm
Location: Waterlooville

Re: Under-voltage event logging?

Wed Feb 07, 2018 4:47 pm

MaxK1 wrote:
Wed Feb 07, 2018 4:26 pm
I think vcgencmd get_throttled only shows that it happened at least once.
I don't know, but I guess Dom does:
dom wrote:
Fri Jul 14, 2017 3:01 pm
If the power drops below the voltage threshold are brief (e.g. a few milliseconds) then you won't spot it on the LED.
But get_throttled has some persistence (matching the visible warning symbol) so it will show any undervoltage events that occurred recently (~ 1 second).
The time constant here suggests how fast to poll for logging <1s.
If you count and report, say, every 10 - 30 seconds, that would give a good indication of the problem.

notro
Posts: 695
Joined: Tue Oct 16, 2012 6:21 pm
Location: Drammen, Norway

Re: Under-voltage event logging?

Sat Feb 17, 2018 4:31 pm


notro
Posts: 695
Joined: Tue Oct 16, 2012 6:21 pm
Location: Drammen, Norway

Re: Under-voltage event logging?

Wed Mar 07, 2018 10:40 am

There is now under-voltage logging in the latest rpi-update:
https://github.com/raspberrypi/linux/pull/2397

Return to “Advanced users”