Is a data logging app possible in Micropython?

[zmarties]

The official Pico book chapter 9 contains a trivially simple temperature logging program in Micropython that works just fine when run on the Pico from the Thonny editor. It's got an infinite loop in it, but when you terminate the program from the editor, the data is safely stored on the disk, and can be retrieved by opening the named file also in the editor.

However, when run on the Pico connected to a battery, I don't get data logged. I assume that is because when I disconnect the battery to reconnect to the host to access the data, the act of cutting the power when the app is still running means that the file system is scrambled. (Occasionally I get a shorter file than I would expect, but most commonly I get no data at all).

Is it even possible to write a reliable data logging app in Micropython, given that the file system seems to be unreliable once power is disconnected?

Note that the logging app already flushes the data on each write, which should mean the data has been committed to disk. I have tried various enhancements to the app, but none of them make any different - the logging always works fine when run from the editor, but as soon as I run the app with the board connected to a battery, the data is not available when I reconnect the board back to the computer. Improvements I have tried include
- flashing the led with each write, so I can be sure that the loop is executing repeatedly
- opening the file (in append mode), writing the data, then closing the file on each time round the loop, rather than just relying on flush
- changing the file to loop a fixed number of times, rather than infinitely.

For the record, I get the same behaviour with various USB batteries, and have tried the app on two different pico boards (one of them is now bricked - but that's a different issue!)

[wilderbeest]

Have you tried to close the file after each read, and to open it again before you write?

[zmarties]

Yes, that's in the list I gave of things I have already tried.

[hippy]

Haven't tried it myself but opening, writing and closing a file should see that persist across power-cycles.

It could of course be a bug if it doesn't.

[zmarties]

Well it certainly looks like it's currently bugged to me.

The program below runs fine direct from Thonny - it flashes the led every second, then when it has written enough data that the file size is greater than 100 bytes it exits. The file can then be opened in Thonny, and the numbers seen.

When run on battery power, the program also flashes the led, and the data must be being written to file, since the same test on the file size causes the program to finish, and stop flashing the led. At this point I disconnect the battery, and connect the lead back to the Windows PC running Thonny. Sometimes there is no data.txt file to be seen, or at other times it is smaller than 100 bytes that it reached or the program would not have exited.

My assumption is that the file system is caching the file (possibly to avoid writing to flash to many times), and when the battery power is removed that data is simply lost.

Code: Select all

from machine import Pin, Timer
import utime
import os

led = Pin(25, Pin.OUT)

filename = "data.txt"
file = open(filename, "w")
file.write("A bunch of times...\n")
file.close()

timer = Timer()

def tick(timer):
    led.toggle()
    time = str(int(utime.ticks_ms() / 1000))
    
    file = open(filename, "a")
    file.write(time + "\n")
    file.close()

    filesize = os.stat(filename)[6]

    if filesize > 100:
        timer.deinit()
        led.value(0);

timer.init(freq=1, mode=Timer.PERIODIC, callback=tick)

[g4bch]

It looks to me as though when you start the program it opens the file in write mode. Does this not erase any existing file with the same name.
Try generating a unique file name at each start up or open the file in append mode.

[zmarties]

It does indeed open in write mode (deliberately to erase any data from a previous run), and then writes a line and closes the file.

When logging the data, it reopens the file in append mode, and writes more data to the file, before closing it again. The stat call is checking that the file size is increasing, and uses that to exit the program when enough data has been written.

[hippy]

When you have done the sampling on battery power, pulled the power, reconnected to your PC; the Pico will boot and start running your program. I'm not sure exactly what happens but I can't see any guarantee that you won't have deleted your sample file and started writing new samples before Thonny connects and issues a Soft Boot to the Pico.

Who knows what state that might leave things in. You might get lucky, get access to the Pico filesystem before it overwrites the old, while overwriting, or when adding samples.

Perhaps adding a time.sleep() at the start of the program will ensure Thonny can can gain control of the Pico before it starts updating the file system ?

My limited testing seemed to show file appending and persistence etc working as expected but I accept that was not exhaustive testing.

[zmarties]

Thanks - that makes sense.

Ideally I need a way to determine whether connected just to power, or to a computer over USB, but I can see that a workaround as you suggest with startup delays might be workable.

[hippy]

I don't know enough about the Pico or MicroPython to know of a way to figure out if it's just powered or has a connection to a Pi/PC over USB - Some sort of 'machine.ConnectedToHost()' or similar would be ideal for that. I expect such things may come as users identify what would be useful to have and others set about adding functionality.

The usual way to handle this issue on other micros is to wait for a button to be pressed before beginning the sampling process. Or only take samples when booted and the file doesn't exist.

[scruss]

While you can log files to the Raspberry Pi Pico's flash, these simple chips have no wear levelling features like SD cards do. Maybe log to an external SD card and keep the built-in flash for program files and libraries that don't change much.

[hippy]

The W25Q16 used on the Pico has a minimum 100,000 write/erase cycles which is probably okay when logging at slow rates, but not great for higher-speed logging.

One could potentially wear out a block in a couple of days by overwriting it once a second. Sequential logging should make it last longer because the same sector isn't being repeatedly hit but repeated flush(), re-creating files, will have more impact. Any file creation - maybe any writing ? - may have a hit on the block holding the directory information.

It is probably best to treat it as read-only when one can.

[DWiskow]

Here is an example temperature logger in Micropython for the Pico Board

UPDATE: code updated to only write to log file when Pico is running on battery power

Code: Select all

import machine
import utime

TEMP_READING_INTERVAL = 5         # number of seconds betweem samples
AVERAGE_READING_COUNT = 12        # number of samples to collect before generating/writing an average
FLUSH_LOG_INTERVAL    = 60 * 15   # flush (write) log file every 15 minutes 

led_onboard = machine.Pin(25, machine.Pin.OUT)
USBpower = machine.Pin(24, machine.Pin.IN)
sensor_temp = machine.ADC(4)
conversion_factor = 3.3 / (65535)

if USBpower() != 1:
    f = open('temp.log', 'a+t')
    f.write('----- temperature Monitor -----'+'\n')
    f.write('==============================='+'\n')
    f.write(str(TEMP_READING_INTERVAL) + 's interval & ' + str(AVERAGE_READING_COUNT) + ' samples average'+'\n')
    f.write('    '+str(FLUSH_LOG_INTERVAL) + 's log flush interval'+'\n')
    f.write('\n')

    def logTemp():
        print ('----- temperature Monitor -----')
        print ('===============================')
        print(str(TEMP_READING_INTERVAL) + 's interval & ' + str(AVERAGE_READING_COUNT) + ' samples average')
        print('    '+str(FLUSH_LOG_INTERVAL) + 's log flush interval')
        print()

        l = 0
        reading = sensor_temp.read_u16() * conversion_factor
        temperature = round(27 - ((reading - 0.706)/0.00172), 1)

        while True:
            if l >= FLUSH_LOG_INTERVAL:
                f.flush()
                l = 0

            f.write(str(temperature)+'\n')
            print('    '+str(temperature)+' - '+str(FLUSH_LOG_INTERVAL-l)+'s until flush')
            
            n = 0
            temperature = 0     
            while n < AVERAGE_READING_COUNT:
                reading = sensor_temp.read_u16() * conversion_factor
                temperature = temperature + (27 - (reading - 0.706)/0.00172)
                n = n + 1
                led_onboard.value(0)
                utime.sleep(TEMP_READING_INTERVAL)
                led_onboard.value(1)
                l = l + TEMP_READING_INTERVAL
                
            temperature = round(temperature / AVERAGE_READING_COUNT, 1)

    try:
        logTemp()
    except KeyboardInterrupt:
        print()
        print('ctrl-c = exit temperature logger')
    finally:
        f.close()

The log file file is opened in ‘append’ mode, so each power up/execution [on battery] will add additional entries to the ‘temp.log’ file (delineated by a ‘title’ block)

This code reads a temperature sample every 5 seconds [and briefly flashes the LED], then collects and averages 12 samples to obtain a stable reading before ‘writing’ the value to a file on the flash . . . but Micropython doesn’t actually write to the flash until the file is either ‘flushed’ or ‘closed’ . . . so, the writes are effectively batched and committed to the flash every 15 minutes in the above code.

To access the data logged on the Pico Board, run Thonny on the computer and connect the Pico. The ‘temp.log’ file can then be opened from within Thonny and the contents viewed/'saved/edited/graphed on the computer, as required. To 'clear down' the log, delete the 'temp.log' file from the Pico by right clicking on it in the ‘file’ window in Thonny and selecting ‘delete’

The code has now been updated to automatically terminate and not log data when powered via USB (i.e. when connected to a computer), and only log temperatures when powered by a battery on VSYS (see post here https://www.raspberrypi.org/forums/vie ... 6&t=300676 for more information on powering the Pico from a battery.

NOTE

• the RD2040 Microcontroller incorporates a Real Time Clock which would be useful for this application AND it is possible to detect a press of the BOOTSEL button (by temporarily disabling interrupts), but this currently requires the application to be written in C/C++

• Have a look at my post https://www.raspberrypi.org/forums/vie ... 9#p1805839 in the RTC topic to see how you can power the Pico from a battery and detect when it is running on battery [rather than plugged into USB power] to determine whether to initiate logging and/or initialise the logging file.

[DWiskow]

Have a look at my post (viewtopic.php?f=146&t=300275#p1807232) on how to implement a 'corrected' RTC in only a few lines of MicroPython for data logging purposes.

[zmarties]

I'd done something similar with inputting a time offset to get a better basis for recording times. However it would be great if MicroPython really implemented a way to set a RTC that the rest of the system would use - especially the filesystem.