zmarties
Posts: 63
Joined: Fri Jan 22, 2021 6:53 pm

Is a data logging app possible in Micropython?

Fri Jan 22, 2021 7:22 pm

The official Pico book chapter 9 contains a trivially simple temperature logging program in Micropython that works just fine when run on the Pico from the Thonny editor. It's got an infinite loop in it, but when you terminate the program from the editor, the data is safely stored on the disk, and can be retrieved by opening the named file also in the editor.

However, when run on the Pico connected to a battery, I don't get data logged. I assume that is because when I disconnect the battery to reconnect to the host to access the data, the act of cutting the power when the app is still running means that the file system is scrambled. (Occasionally I get a shorter file than I would expect, but most commonly I get no data at all).

Is it even possible to write a reliable data logging app in Micropython, given that the file system seems to be unreliable once power is disconnected?

Note that the logging app already flushes the data on each write, which should mean the data has been committed to disk. I have tried various enhancements to the app, but none of them make any different - the logging always works fine when run from the editor, but as soon as I run the app with the board connected to a battery, the data is not available when I reconnect the board back to the computer. Improvements I have tried include
- flashing the led with each write, so I can be sure that the loop is executing repeatedly
- opening the file (in append mode), writing the data, then closing the file on each time round the loop, rather than just relying on flush
- changing the file to loop a fixed number of times, rather than infinitely.

For the record, I get the same behaviour with various USB batteries, and have tried the app on two different pico boards (one of them is now bricked - but that's a different issue!)

wilderbeest
Posts: 4
Joined: Fri Jan 22, 2021 8:03 pm

Re: Is a data logging app possible in Micropython?

Sat Jan 23, 2021 12:00 pm

Have you tried to close the file after each read, and to open it again before you write?

zmarties
Posts: 63
Joined: Fri Jan 22, 2021 6:53 pm

Re: Is a data logging app possible in Micropython?

Sat Jan 23, 2021 3:52 pm

Yes, that's in the list I gave of things I have already tried.

hippy
Posts: 9132
Joined: Fri Sep 09, 2011 10:34 pm
Location: UK

Re: Is a data logging app possible in Micropython?

Sat Jan 23, 2021 4:00 pm

Haven't tried it myself but opening, writing and closing a file should see that persist across power-cycles.

It could of course be a bug if it doesn't.

zmarties
Posts: 63
Joined: Fri Jan 22, 2021 6:53 pm

Re: Is a data logging app possible in Micropython?

Sat Jan 23, 2021 5:55 pm

Well it certainly looks like it's currently bugged to me.

The program below runs fine direct from Thonny - it flashes the led every second, then when it has written enough data that the file size is greater than 100 bytes it exits. The file can then be opened in Thonny, and the numbers seen.

When run on battery power, the program also flashes the led, and the data must be being written to file, since the same test on the file size causes the program to finish, and stop flashing the led. At this point I disconnect the battery, and connect the lead back to the Windows PC running Thonny. Sometimes there is no data.txt file to be seen, or at other times it is smaller than 100 bytes that it reached or the program would not have exited.

My assumption is that the file system is caching the file (possibly to avoid writing to flash to many times), and when the battery power is removed that data is simply lost.

Code: Select all

from machine import Pin, Timer
import utime
import os

led = Pin(25, Pin.OUT)

filename = "data.txt"
file = open(filename, "w")
file.write("A bunch of times...\n")
file.close()

timer = Timer()

def tick(timer):
    led.toggle()
    time = str(int(utime.ticks_ms() / 1000))
    
    file = open(filename, "a")
    file.write(time + "\n")
    file.close()

    filesize = os.stat(filename)[6]

    if filesize > 100:
        timer.deinit()
        led.value(0);

timer.init(freq=1, mode=Timer.PERIODIC, callback=tick)

g4bch
Posts: 13
Joined: Mon Oct 01, 2012 12:06 pm

Re: Is a data logging app possible in Micropython?

Sat Jan 23, 2021 6:16 pm

It looks to me as though when you start the program it opens the file in write mode. Does this not erase any existing file with the same name.
Try generating a unique file name at each start up or open the file in append mode.

zmarties
Posts: 63
Joined: Fri Jan 22, 2021 6:53 pm

Re: Is a data logging app possible in Micropython?

Sat Jan 23, 2021 6:23 pm

It does indeed open in write mode (deliberately to erase any data from a previous run), and then writes a line and closes the file.

When logging the data, it reopens the file in append mode, and writes more data to the file, before closing it again. The stat call is checking that the file size is increasing, and uses that to exit the program when enough data has been written.

hippy
Posts: 9132
Joined: Fri Sep 09, 2011 10:34 pm
Location: UK

Re: Is a data logging app possible in Micropython?

Sun Jan 24, 2021 11:25 am

When you have done the sampling on battery power, pulled the power, reconnected to your PC; the Pico will boot and start running your program. I'm not sure exactly what happens but I can't see any guarantee that you won't have deleted your sample file and started writing new samples before Thonny connects and issues a Soft Boot to the Pico.

Who knows what state that might leave things in. You might get lucky, get access to the Pico filesystem before it overwrites the old, while overwriting, or when adding samples.

Perhaps adding a time.sleep() at the start of the program will ensure Thonny can can gain control of the Pico before it starts updating the file system ?

My limited testing seemed to show file appending and persistence etc working as expected but I accept that was not exhaustive testing.

zmarties
Posts: 63
Joined: Fri Jan 22, 2021 6:53 pm

Re: Is a data logging app possible in Micropython?

Sun Jan 24, 2021 11:37 am

Thanks - that makes sense.

Ideally I need a way to determine whether connected just to power, or to a computer over USB, but I can see that a workaround as you suggest with startup delays might be workable.

hippy
Posts: 9132
Joined: Fri Sep 09, 2011 10:34 pm
Location: UK

Re: Is a data logging app possible in Micropython?

Sun Jan 24, 2021 12:24 pm

I don't know enough about the Pico or MicroPython to know of a way to figure out if it's just powered or has a connection to a Pi/PC over USB - Some sort of 'machine.ConnectedToHost()' or similar would be ideal for that. I expect such things may come as users identify what would be useful to have and others set about adding functionality.

The usual way to handle this issue on other micros is to wait for a button to be pressed before beginning the sampling process. Or only take samples when booted and the file doesn't exist.

User avatar
scruss
Posts: 3794
Joined: Sat Jun 09, 2012 12:25 pm
Location: Toronto, ON
Contact: Website

Re: Is a data logging app possible in Micropython?

Sun Jan 24, 2021 3:00 pm

While you can log files to the Raspberry Pi Pico's flash, these simple chips have no wear levelling features like SD cards do. Maybe log to an external SD card and keep the built-in flash for program files and libraries that don't change much.
‘Remember the Golden Rule of Selling: “Do not resort to violence.”’ — McGlashan.
Pronouns: he/him

hippy
Posts: 9132
Joined: Fri Sep 09, 2011 10:34 pm
Location: UK

Re: Is a data logging app possible in Micropython?

Sun Jan 24, 2021 3:48 pm

The W25Q16 used on the Pico has a minimum 100,000 write/erase cycles which is probably okay when logging at slow rates, but not great for higher-speed logging.

One could potentially wear out a block in a couple of days by overwriting it once a second. Sequential logging should make it last longer because the same sector isn't being repeatedly hit but repeated flush(), re-creating files, will have more impact. Any file creation - maybe any writing ? - may have a hit on the block holding the directory information.

It is probably best to treat it as read-only when one can.

DWiskow
Posts: 54
Joined: Sun Apr 09, 2017 1:56 pm

Re: Is a data logging app possible in Micropython?

Tue Jan 26, 2021 1:02 am

Here is an example temperature logger in Micropython for the Pico Board

UPDATE: code updated to only write to log file when Pico is running on battery power

Code: Select all

import machine
import utime

TEMP_READING_INTERVAL = 5         # number of seconds betweem samples
AVERAGE_READING_COUNT = 12        # number of samples to collect before generating/writing an average
FLUSH_LOG_INTERVAL    = 60 * 15   # flush (write) log file every 15 minutes 

led_onboard = machine.Pin(25, machine.Pin.OUT)
USBpower = machine.Pin(24, machine.Pin.IN)
sensor_temp = machine.ADC(4)
conversion_factor = 3.3 / (65535)

if USBpower() != 1:
    f = open('temp.log', 'a+t')
    f.write('----- temperature Monitor -----'+'\n')
    f.write('==============================='+'\n')
    f.write(str(TEMP_READING_INTERVAL) + 's interval & ' + str(AVERAGE_READING_COUNT) + ' samples average'+'\n')
    f.write('    '+str(FLUSH_LOG_INTERVAL) + 's log flush interval'+'\n')
    f.write('\n')

    def logTemp():
        print ('----- temperature Monitor -----')
        print ('===============================')
        print(str(TEMP_READING_INTERVAL) + 's interval & ' + str(AVERAGE_READING_COUNT) + ' samples average')
        print('    '+str(FLUSH_LOG_INTERVAL) + 's log flush interval')
        print()

        l = 0
        reading = sensor_temp.read_u16() * conversion_factor
        temperature = round(27 - ((reading - 0.706)/0.00172), 1)

        while True:
            if l >= FLUSH_LOG_INTERVAL:
                f.flush()
                l = 0

            f.write(str(temperature)+'\n')
            print('    '+str(temperature)+' - '+str(FLUSH_LOG_INTERVAL-l)+'s until flush')
            
            n = 0
            temperature = 0     
            while n < AVERAGE_READING_COUNT:
                reading = sensor_temp.read_u16() * conversion_factor
                temperature = temperature + (27 - (reading - 0.706)/0.00172)
                n = n + 1
                led_onboard.value(0)
                utime.sleep(TEMP_READING_INTERVAL)
                led_onboard.value(1)
                l = l + TEMP_READING_INTERVAL
                
            temperature = round(temperature / AVERAGE_READING_COUNT, 1)

    try:
        logTemp()
    except KeyboardInterrupt:
        print()
        print('ctrl-c = exit temperature logger')
    finally:
        f.close()

The log file file is opened in ‘append’ mode, so each power up/execution [on battery] will add additional entries to the ‘temp.log’ file (delineated by a ‘title’ block)

This code reads a temperature sample every 5 seconds [and briefly flashes the LED], then collects and averages 12 samples to obtain a stable reading before ‘writing’ the value to a file on the flash . . . but Micropython doesn’t actually write to the flash until the file is either ‘flushed’ or ‘closed’ . . . so, the writes are effectively batched and committed to the flash every 15 minutes in the above code.

To access the data logged on the Pico Board, run Thonny on the computer and connect the Pico. The ‘temp.log’ file can then be opened from within Thonny and the contents viewed/'saved/edited/graphed on the computer, as required. To 'clear down' the log, delete the 'temp.log' file from the Pico by right clicking on it in the ‘file’ window in Thonny and selecting ‘delete’

The code has now been updated to automatically terminate and not log data when powered via USB (i.e. when connected to a computer), and only log temperatures when powered by a battery on VSYS (see post here https://www.raspberrypi.org/forums/vie ... 6&t=300676 for more information on powering the Pico from a battery.

NOTE

  • the RD2040 Microcontroller incorporates a Real Time Clock which would be useful for this application AND it is possible to detect a press of the BOOTSEL button (by temporarily disabling interrupts), but this currently requires the application to be written in C/C++
  • Have a look at my post https://www.raspberrypi.org/forums/vie ... 9#p1805839 in the RTC topic to see how you can power the Pico from a battery and detect when it is running on battery [rather than plugged into USB power] to determine whether to initiate logging and/or initialise the logging file.
Last edited by DWiskow on Fri Jan 29, 2021 11:57 am, edited 5 times in total.

DWiskow
Posts: 54
Joined: Sun Apr 09, 2017 1:56 pm

Re: Is a data logging app possible in Micropython?

Thu Jan 28, 2021 11:28 pm

Have a look at my post (viewtopic.php?f=146&t=300275#p1807232) on how to implement a 'corrected' RTC in only a few lines of MicroPython for data logging purposes.

zmarties
Posts: 63
Joined: Fri Jan 22, 2021 6:53 pm

Re: Is a data logging app possible in Micropython?

Fri Jan 29, 2021 11:33 am

I'd done something similar with inputting a time offset to get a better basis for recording times. However it would be great if MicroPython really implemented a way to set a RTC that the rest of the system would use - especially the filesystem.

Return to “MicroPython”